The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review
Open accessThe DOI matches its Crossref record; 210 of the 211 references it cites were found in a registry or an index; 44 of the 77 sources its claims cite were checked for support.
Abstract
The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. The development of trustworthy AI is especially important in medicine due to the large implications for patients’ lives. While trustworthiness concerns various aspects including ethical, transparency and safety requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of medical ML products. We perform a systematic review following PRISMA guidelines using the databases Web of Science, PubMed and ACM Digital Library. We identify 5408 studies, out of which 120 records fulfil our eligibility criteria. From this literature, we synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions, along which developers of medical ML applications should investigate the content of a dataset. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine. The METRIC-framework may serve as a base for systematically assessing training datasets, establishing reference datasets, and designing test datasets which has the potential to accelerate the approval of medical ML products.
Overview
CoChat Summary
TL;DR
This systematic review synthesizes data quality literature and proposes the METRIC-framework, a set of 15 awareness dimensions for assessing medical training data quality for trustworthy AI. It is intended to help evaluate whether a dataset’s content is appropriate for a specific medical ML use case.
Key findings
- From 5408 identified studies (4633 from database search, 775 from snowballing), 120 records met eligibility criteria; the corpus comprised 35 general data, 8 big data, and 77 ML data entries, with 76 non-life science and 44 life science studies.
- The authors extracted 461 different data quality terms with 991 mentions and hierarchically clustered them into 38 relevant dimensions/subdimensions, organized into 15 awareness dimensions across five clusters: measurement process, timeliness, representativeness, informativeness, and consistency.
- The METRIC-framework separates data management (documentation, security, privacy, FAIR principles) as a prerequisite rather than part of content quality; the dimensions are not yet ranked or equipped with measures.
- The review recommends prioritizing six dimensions for assessment/improvement: the representativeness cluster plus feature importance, distribution consistency, and human-induced error; ML papers were skewed toward robustness, amount of data, missing data, and label noise, with fairness and explainability underrepresented.
Methods
- Unregistered systematic review following PRISMA guidelines; searched Web of Science, PubMed, and ACM Digital Library on 12 April 2024 using a predefined string combining data quality terms with framework/dimensions or ML/AI/DL terms.
- Title/abstract screening was done by two authors independently, with consensus discussion and a third author if needed; snowballing added 775 records; full-text assessment yielded 120 included records.
- Data extraction: 120 records were read by two authors, relevant terms extracted, and related concepts clustered into subdimensions requiring at least three references; definitions were adopted from Black and van Nederpelt, Wang and Strong, or derived from the corpus.
Limitations
- The paper states the review was unregistered; database choice may skew interpretations somewhat, mitigated by snowballing; results are limited to ML approaches despite using “AI” terminology; the framework does not yet include assessment/measurement guidelines, dimensions are not ranked, and literature may be skewed toward dimensions with larger reported effects.
- Evident from the text: the framework is conceptual and not empirically validated, and the authors note that quantification of data quality dimension effects on ML applications is future work.
- Evident from the text: no inter-rater reliability statistics are reported for screening or extraction, although the process used independent reviewers and consensus.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
23 Valid · 2 Not valid · 7 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another dominant topic is completeness to which the ML community almost exclusively refers to as missing data119,125–128,133–135,182.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Results > ML data quality
What the sources say
- Partly supportedcites [119] Jouseau, R., Salva, S. & Samir, C. On studying the effect of data quality on classification performances. Intelligent Data Engineering and Automated Learning – IDEAL. 82–93 (Springer Cham, 2022). sourceExperimental — accuracy not yet measured
The Introduction (p. 3) identifies “missing values” as a data-quality error studied in machine-learning classification, so this paper exemplifies that terminology. It does not call the topic “completeness” or establish that the ML community uses “missing data” almost exclusively; nor does it establish the topic’s dominance in DL papers. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [125] Michel, E., Zernikow, B. & Wichert, S. A. Use of an artificial neural network (ANN) for classifying nursing care needed, using incomplete input data. Med. Inform. Internet Med. 25, 147–158 (2000). 10901277 10.1080/14639230050058329 source
- Supportedcites [126] Barakat, M. S. et al. The effect of imputing missing clinical attribute values on training lung cancer survival prediction model performance. Health Inf. Sci. Syst.5, 16 (2017). PMC5718991 29255599 source
The Introduction discusses missing data in machine-learning datasets and contrasts records with missing data against complete sets. The Conclusion likewise contrasts a complete dataset with levels of missing data. This supports the terminology for which this paper is cited; whether completeness is a dominant topic is the citing authors’ own framing. (LLM, conf=0.86)
- Supportedcites [127] Radliński, Ł. The impact of data quality on software testing effort prediction. Electronics12, 1656 (2023). source
The Introduction (page unavailable) frames missing values as a data-quality problem for machine-learning models and discusses ways to handle them. The paper therefore supports its use as an example of ML work on missing data; the broader characterization of the literature is the citing authors’ synthesis. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [128] Ghotra, B., McIntosh, S. & Hassan, A. E. Revisiting the impact of classification techniques on the performance of defect prediction models. In Proc. IEEE/ACM 37th IEEE International Conference on Software Engineering (2015). source
- Not checked — no full text of the sourcecites [133] Blake, R. & Mangiameli, P. The effects and interactions of data quality and problem complexity on classification. J. Data Inf. Qual. 2, 1–28 (2011). 10.1145/1891879.1891881 source
- Partly supportedcites [134] Benedick, P.-L., Robert, J. & Traon, Y. L. A systematic approach for evaluating artificial intelligence models in industrial settings. Sensors 21, 6195 (2021). 34577398 10.3390/s21186195 PMC8469892 sourceExperimental — accuracy not yet measured
Section 3 describes missing time-series data points as a “dropping perturbation” and discusses filling missing values. The paper does not say that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively calls it “missing data”; its literature review in Section 2 instead focuses on robustness under perturbations. (LLM, conf=0.82)
- Supportedcites [135] Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8, 6085 (2018). 29666385 10.1038/s41598-018-24271-9 PMC5904216 source
The Introduction describes a deep-learning framework for time series with missing data, and the Summary describes handling missing values with recurrent neural networks. This paper therefore serves as an example of the deep-learning literature using “missing data” and “missing values” for the issue the citing sentence calls completeness. (LLM, conf=0.85)
- Partly supportedcites [182] Whang, S. E., Roh, Y., Song, H. & Lee, J.-G. Data collection and quality challenges in deep learning: a data-centric AI perspective. VLDB J. 32, 791–813 (2023). 10.1007/s00778-022-00775-9 sourceExperimental — accuracy not yet measured
Section 4.2 (p. 10) says missing-data imputation is an active research topic in statistics and machine learning, supporting the claim that missing data is a significant topic. The paper does not establish that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively uses “missing data” to refer to completeness. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If the data used for training a model is bad, the resulting AI will be bad as well (‘garbage in, garbage out’49).”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Supportedcites [49] Geiger, R. S. et al. Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from? In Proc. Conference on Fairness, Accountability, and Transparency, 325–336 (2020). source
The Introduction (p. 1) says supervised machine learning requires high-quality training data to produce high-quality classifiers. It also explicitly explains ‘Garbage In, Garbage Out’ as the principle that flawed inputs produce flawed outputs, supporting the citing sentence’s paraphrase. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Neural networks are prone to learning biases from training data and amplifying them at test time50, giving rise to a much discussed aspect of AI behaviour: fairness51.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Partly supportedcites [50] Zhao, J., Wang, T., Yatskar, M., Ordonez, V. & Chang, K.-W. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proc. Conference on Empirical Methods in Natural Language Processing, 2979–2989 (Association for Computational Linguistics, Copenhagen, Denmark, 2017). sourceExperimental — accuracy not yet measured
Section 6.3 reports that models trained on two gender-biased image datasets amplified those biases on development data. The Model and §5.2 passages show that these were structured prediction models using convolutional neural-network features. This supports the claimed phenomenon in those systems, but not the broader assertion that neural networks generally are prone to it. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet, one of the main causes of undesirable learned patterns lies in biased training data55,56.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Partly supportedcites [55] Suresh, H. & Guttag, J. A framework for understanding sources of harm throughout the machine learning life cycle. In Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’21), October 5–9, 2021, NY, USA. ACM, New York, NY, USA. 10.1145/3465416.3483305 (2021). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) acknowledges that biased data can contribute to undesirable ML behavior, and §3.2 describes how an unrepresentative development sample can impair a model. The paper does not rank biased training data as a main cause; it also identifies sources of harm in modeling, evaluation, and deployment. (LLM, conf=0.85)
- Supportedcites [56] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. 54, 1–35 (2021). 10.1145/3457607 source
Section 3, “Bias in Data, Algorithms, and User Experiences,” says that algorithms trained on biased data learn those biases and reflect them in their predictions. It also recognizes other causes of bias, which is consistent with describing biased training data as one cause rather than the only cause. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In particular, representativeness is often not a factor86,87 while it is a crucial property for secondary use of data in clinical studies88 or when reusing medical data as training data for ML applications.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Results > General data quality
What the sources say
- Not checked — no full text of the sourcecites [86] Chan, K. S., Fowles, J. B. & Weiner, J. P. Electronic health records and the reliability and validity of quality measures: a review of the literature. Med. Care Res. Rev. 67, 503–527 (2010). 20150441 10.1177/1077558709359007 source
- Supportedcites [87] Weiskopf, N. G. & Weng, C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J. Am. Med. Inform. Assoc. 20, 144–151 (2013). 22733976 10.1136/amiajnl-2011-000681 PMC3555312 source
The Results section identifies five dimensions of EHR data quality derived from the reviewed literature: completeness, correctness, concordance, plausibility, and currency. Representativeness is not among them, supporting the claim that it is often absent as a data-quality factor. The indexed copy provides no page number for this passage. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
22 more claims · 4 with their cited sources checked
Create a free account to see all 102 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 31
- Probably not a substantial claim. 61
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“During the last decade, the fi eld of arti fi cial intelligence (AI) and in particular machine learning (ML) has experienced unprecedented advances, largely due to breakthroughs in deep learning (DL) 1 – 5 and increased computational power.”
BackgroundU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 1×
“During the last decade, the fi eld of arti fi cial intelligence (AI) and in particular machine learning (ML) has experienced unprecedented advances, largely due to breakthroughs in deep learning (DL) 1 – 5 and increased computational power.”
BackgroundHighly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“Recently, the introduction of easy-to-use yet still extremely capable models such as GPT-4 6 and Stable Diffusion 7 has further expanded the technology to an even broader audience.”
MethodThe FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 2×
“However, theracefor implementation and utilisation is currently outpacing comprehension of the technology.”
BackgroundA survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“Che et al. 135 fi nd aninformative patterninthecase of theMIMIC-III critical care dataset 226 whichdisplaysacorrelationbetweenmissingratesofvariablesand ICD9-diagnosislabels.”
BackgroundGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Despite breakthroughs up to human-level performance 9,18 – 20 , ML-backed medical products are mainly used as diagnosis assistance systems 17 leaving the fi nal decision to medical human professionals.”
BackgroundExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781 · cited 1×
“Among expert annotators there is often considerable variability 206,207 .”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Several approaches and de fi nitions for trustworthy AI have been discussed and published over the past years by researchers 29 – 33 , public entities 34,35 , corporations 36 , and organisations 37,38 .”
BackgroundFashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108 · cited 1×
“Even in the most common (non-medical) datasets of ML (e.g., MNIST 208 , CIFAR-100 209 , Fashion-MNIST 210 ) there is a signi fi cant percentage of wrong labels 211,212 .”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 1×
“These standardisation efforts are a crucial fi rst step for developing a better understanding of ML systems as a whole and of the interdependence of its components (e.g., data and algorithm).”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812 · cited 1×
“These standardisation efforts are a crucial fi rst step for developing a better understanding of ML systems as a whole and of the interdependence of its components (e.g., data and algorithm).”
BackgroundThe MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
Deng, L. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2211477 · Cited by 4,836 · cited 1×
“Even in the most common (non-medical) datasets of ML (e.g., MNIST 208 , CIFAR-100 209 , Fashion-MNIST 210 ) there is a signi fi cant percentage of wrong labels 211,212 .”
BackgroundBeyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672 · cited 6×
“The overarching topics Theearly data quality research in the 1980s and 1990s uncovered the lackofobjectivemeasurestoassesdataquality,whichledtotheintroduction oftask-dependentdimensionsandtheestablishmentofadataqualityfra-meworkfromtheperspectiveofthedataconsumer 57 .”
BackgroundMethodInfluentialTransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“Finally, the highly-discussed dimension target class balance pays tri-bute to the technical requirements of ML 140,141,144,150,159 .”
BackgroundSurvey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474 · cited 4×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundInfluentialGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 4×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundInfluentialRecurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-08999-2_22 · Cited by 2,273
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883 · cited 1×
“Despite breakthroughs up to human-level performance 9,18 – 20 , ML-backed medical products are mainly used as diagnosis assistance systems 17 leaving the fi nal decision to medical human professionals.”
BackgroundClassification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 1×
“Each record which had passed the screening was eligible for extracting data quality terms.”
MethodTraining with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 3×
“In particular, representativeness is often not a factor 86,87 while it is a crucial property for secondary use of data in clinical studies 88 or when reusing medical data as training data for ML applications.”
BackgroundInfluentialThe Challenges of Data Quality and Data Quality Assessment in the Big Data Era
Cai, L., & Zhu, Y. (2015) · Data Science Journal · doi:10.5334/dsj-2015-002 · Cited by 1,010 · cited 1×
“Further, the increasing amount of data requires the computational ef fi ciency of the surrounding database infrastructure to be a part of big data quality frameworks 108 – 110 .”
BackgroundCan You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
IEEE Standard Glossary of Software Engineering Terminology
(2008) · doi:10.1109/ieeestd.1990.101064 · Cited by 979
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundUnderstanding how image quality affects deep neural networks
Dodge, S., & Karam, L. (2016) · 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2016.7498955 · Cited by 835
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“…40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset Nutrition Label 45 , Data Cards 46 , Healthsheet 47 , Data Statements for NLP 48 ).”
BackgroundKhatri, V., & Brown, C. V. (2010) · Communications of the ACM · doi:10.1145/1629175.1629210 · Cited by 816 · cited 1×
“The initial search of the databases Web of Science, PubMed and ACM Digital Library was performed on the 12th of April 2024 and yielded 4633 unique results.”
BackgroundConfident Learning: Estimating Uncertainty in Dataset Labels
Northcutt, C., Jiang, L., & Chuang, I. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12125 · Cited by 795
Transparency and the black box problem: why we do not trust AI
von Eschenbach, W. J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00477-0 · Cited by 786 · cited 1×
“A lack of con fi dence of the general population in the transparency of AI prevents its utilisation for society and economic growth.”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · cited 1×
“Recently, the introduction of easy-to-use yet still extremely capable models such as GPT-4 6 and Stable Diffusion 7 has further expanded the technology to an even broader audience.”
MethodDeep learning with noisy labels: Exploring techniques and remedies in medical image analysis
Karimi, D., Dou, H., Warfield, S. K., & Gholipour, A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101759 · Cited by 706
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 697
A Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
Muehlematter, U. J., Daniore, P., & Vokinger, K. N. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30292-2 · Cited by 665 · cited 1×
“Despite breakthroughs up to human-level performance 9,18 – 20 , ML-backed medical products are mainly used as diagnosis assistance systems 17 leaving the fi nal decision to medical human professionals.”
Background(2001) · PubMed · Cited by 612
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
A Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576
Data collection and quality challenges in deep learning: a data-centric AI perspective
Whang, S. E., Roh, Y., Song, H., & Lee, J.-G. (2023) · The VLDB Journal · doi:10.1007/s00778-022-00775-9 · Cited by 573 · cited 3×
“…look at data quality in the context of sequential data (often time series) 133 – 139 , images 119, 139 – 182 , natural language 120, 121 , 145,182 – 186 or other complex types When it comes to the concrete behaviour change of the DL algorithm, most of the DL papers in the literature corpus…”
BackgroundMethodTraining Convolutional Networks with Noisy Labels
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L. D., & Fergus, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2080 · Cited by 443
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · cited 3×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING…”
BackgroundHernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410 · cited 2×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING…”
BackgroundImpact of Dataset Size on Classification Performance: An Empirical Evaluation in the Medical Domain
Althnian, A., AlSaeed, D., Al-Baity, H., Samha, A., Dris, A. B., Alzakari, N., Abou Elwafa, A., & Kurdi, H. (2021) · Applied Sciences · doi:10.3390/app11020796 · Cited by 410
Deep Learning is Robust to Massive Label Noise
Rolnick, D., Veit, A., Belongie, S., & Shavit, N. (2017) · arXiv · doi:10.48550/arxiv.1705.10694 · Cited by 402
Chan, K. S., Fowles, J. B., & Weiner, J. P. (2010) · Medical Care Research and Review · doi:10.1177/1077558709359007 · Cited by 365
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
A Review of Data Quality Assessment Methods for Public Health Information Systems
Chen, H., Hailey, D., Wang, N., & Yu, P. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110505170 · Cited by 354
Learning Not to Learn: Training Deep Neural Networks With Biased Data
Kim, B., Kim, H., Kim, K., Kim, S., & Kim, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00922 · Cited by 345
A framework for information quality assessment
Stvilia, B., Gasser, L., Twidale, M. B., & Smith, L. C. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20652 · Cited by 339 · cited 1×
“In the following years, the frameworks were further extended 76,77 , equipped with measures 78,79 and re fi ned 80,81 .”
BackgroundData-centric Artificial Intelligence: A Survey
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2025) · ACM Computing Surveys · doi:10.1145/3711118 · Cited by 334
Kim, W., Choi, B.-J., Hong, E.-K., Kim, S.-K., & Lee, D. (2003) · Data Mining and Knowledge Discovery · doi:10.1023/a:1021564703268 · Cited by 301 · cited 1×
“However, human decision making can be highly irrational and subjective, especially in the medical context 203 – 205 , representing one of various sources of labelling noise 206 .”
BackgroundScientific Standards in Epidemiologic Studies of the Menace of Daily Life
Feinstein, A. R. (1988) · Science · doi:10.1126/science.3057627 · Cited by 297 · cited 1×
“Since then, data integrity (de fi ned by the ALCOA-principles or ALCOA + 25 ) has become an essential requirement of several guidelines, such as good clinical practice 26 , good laboratory practice 27 or good manufacturing practice 28 .”
The value of standards for health datasets in artificial intelligence-based applications
Arora, A., Alderman, J. E., Palmer, J., Ganapathi, S., Laws, E., McCradden, M. D., Oakden-Rayner, L., Pfohl, S. R., Ghassemi, M., McKay, F., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02608-w · Cited by 294
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00894 · Cited by 274
Enhancing data quality in data warehouse environments
Ballou, D. P., & Tayi, G. K. (1999) · Communications of the ACM · doi:10.1145/291469.291471 · Cited by 269 · cited 1×
“After title and abstract screening, adding references of the remaining records ( ‘ snowballing ’ ) and full text assessment, we fi nd 120 records that match our eligibility criteria (see Methods).”
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
Effects of Image Degradation and Degradation Removal to CNN-Based Image Classification
Pei, Y., Huang, Y., Zou, Q., Zhang, X., & Wang, S. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950923 · Cited by 253
Bailly, A., Blanc, C., Francis, É., Guillotin, T., Jamal, F., Wakim, B., & Roy, P. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106504 · Cited by 252
Learning with Noisy Labels Revisited: A Study Using Real-World Human AnnotationsPreprint
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12088 · Cited by 243
Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI
Pushkarna, M., Zaldivar, A., & Kjartansson, O. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533231 · Cited by 222 · cited 2×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundData quality: A survey of data quality dimensions
Sidi, F., Shariat Panahy, P. H., Affendey, L. S., Jabar, M. A., Ibrahim, H., & Mustapha, A. (2012) · 2012 International Conference on Information Retrieval & Knowledge Management · doi:10.1109/infrkm.2012.6204995 · Cited by 214
The Devil of Face Recognition Is in the Noise
Wang, F., Chen, L., Li, C., Huang, S., Chen, Y., Qian, C., & Loy, C. C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_47 · Cited by 201
Human Uncertainty Makes Classification More Robust
Peterson, J., Battleday, R., Griffiths, T., & Russakovsky, O. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00971 · Cited by 196
Rage against the machine? Google’s self-driving cars versus human drivers
Teoh, E. R., & Kidd, D. G. (2017) · Journal of Safety Research · doi:10.1016/j.jsr.2017.08.008 · Cited by 190 · cited 1×
“ML is proving to have great potential toimprove many aspectsof ourlives 9 – 11 .”
BackgroundData quality issues in implementing an ERP
Xu, H., Horn Nord, J., Brown, N., & Daryl Nord, G. (2002) · Industrial Management & Data Systems · doi:10.1108/02635570210414668 · Cited by 182 · cited 1×
“For the dimension device error , we distinguish between the subdimension accuracy , the systematic deviation from the ground truth (also called bias), and the sub-dimension precision , the variance of the data around a mean value (also called noise).”
MethodDeng, L. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2762725 · Cited by 180
Electronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171
(2019) · doi:10.1145/3351095.3372862 · Cited by 167 · cited 1×
“…development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset Nutrition Label 45 , Data Cards…”
BackgroundSibony, O., Paris, H., Kahneman, D., & Princeton University (2021) · Character Lab Playbook · doi:10.53776/playbooks-judgment · Cited by 159
Towards Risk‐Free Trustworthy Artificial Intelligence: Significance and Requirements
Alzubaidi, L., Al-Sabaawi, A., Bai, J., Dukhan, A., Alkenani, A. H., Al‐Asadi, A., Alwzwazy, H. A., Manoufali, M., Fadhel, M. A., Albahri, A. S., et al. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/4459198 · Cited by 152 · cited 1×
“Several approaches and de fi nitions for trustworthy AI have been discussed and published over the past years by researchers 29 – 33 , public entities 34,35 , corporations 36 , and organisations 37,38 .”
BackgroundNoise Injection: Theoretical Prospects
Grandvalet, Y., Canu, S., & Boucheron, S. (1997) · Neural Computation · doi:10.1162/neco.1997.9.5.1093 · Cited by 147
Geiger, R. S., Cope, D., Ip, J., Lotosh, M., Shah, A., Weng, J., & Tang, R. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00144 · Cited by 143
DataPerf: Benchmarks for Data-Centric AI Development
Mazumder, M., Banbury, C., Yao, X., Karlaš, B., Rojas, W. G., Diamos, S., Diamos, G., He, L., Parrish, A., Kirk, H. R., et al. (2022) · TU/e Research Portal · doi:10.48550/arxiv.2207.10062 · Cited by 138
Adversarial Attacks on Deep Neural Networks for Time Series Classification
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851936 · Cited by 136
From Data Quality to Big Data Quality
Batini, C., Rula, A., Scannapieco, M., & Viscusi, G. (2015) · Journal of Database Management · doi:10.4018/jdm.2015010103 · Cited by 135
Pooch, E. H. P., Ballester, P., & Barros, R. C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-62469-9_7 · Cited by 134
The Effects and Interactions of Data Quality and Problem Complexity on Classification
Blake, R., & Mangiameli, P. (2011) · Journal of Data and Information Quality · doi:10.1145/1891879.1891881 · Cited by 132
Zhu, S., Gilbert, M., Chetty, I., & Siddiqui, F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104828 · Cited by 131 · cited 2×
“The amount of ML solutions in medicine (research tools and commercial products) is steadily on the rise, in particular in the fi elds of radiology and cardiology 16,17 .”
BackgroundSchmidt, C. O., Struckmann, S., Enzenbach, C., Reineke, A., Stausberg, J., Damerow, S., Huebner, M., Schmidt, B., Sauerbrei, W., & Richter, A. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01252-7 · Cited by 126
Lim, L., Seubsman, S.-A., & Sleigh, A. (2008) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-6-52 · Cited by 123 · cited 1×
“We also omit records discussingcasestudiesofsurveydataquality 65,66 ,aswellastrainingstrategies to cope with bad data 67 – 72 .”
MethodDigital Health Data Quality Issues: Systematic Review
Syed, R., Eden, R., Makasi, T., Chukwudi, I., Mamudu, A., Kamalpour, M., Kapugama Geeganage, D., Sadeghianasl, S., Leemans, S. J. J., Goel, K., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/42615 · Cited by 120
Bansal, A., Kauffman, R. J., & Weitz, R. R. (1993) · Journal of Management Information Systems · doi:10.1080/07421222.1993.11517988 · Cited by 112
Data quality in ETL process: A preliminary study
Souibgui, M., Atigui, F., Zammali, S., Cherfi, S. S.-S., & Yahia, S. B. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.09.223 · Cited by 111 · cited 1×
“This concerns aspects such as data integration 60 , information quality management 61 , ETL processes in data warehouses 62 , or tools for data warehouses 63,64 which do not affect the behavioural characteristics of AI systems.”
BackgroundGuo, L. L., Pfohl, S. R., Fries, J., Johnson, A. E. W., Posada, J., Aftandilian, C., Shah, N., & Sung, L. (2022) · Scientific Reports · doi:10.1038/s41598-022-06484-1 · Cited by 103 · cited 2×
“3), which is based on our literature corpus 57, 74 – 192 .”
BackgroundThe Dataset Nutrition Label: A Framework to Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2020) · Data Protection and Privacy · doi:10.5040/9781509932771.ch-001 · Cited by 103
Training Strategies for Radiology Deep Learning Models in Data-limited Scenarios
Candemir, S., Nguyen, X. V., Folio, L. R., & Prevedello, L. M. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021210014 · Cited by 102 · cited 1×
“We also omit records discussingcasestudiesofsurveydataquality 65,66 ,aswellastrainingstrategies to cope with bad data 67 – 72 .”
MethodData-centric Artificial Intelligence: A SurveyPreprint
Zha, D., Bhat, Z. P., Lai, K.-H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.10158 · Cited by 100
How Image Degradations Affect Deep CNN-Based Face Recognition?
Karahan, S., Kilinc Yildirum, M., Kirtac, K., Rende, F. S., Butun, G., & Ekenel, H. K. (2016) · 2016 International Conference of the Biometrics Special Interest Group (BIOSIG) · doi:10.1109/biosig.2016.7736924 · Cited by 99
Big Data Validation and Quality Assurance -- Issuses, Challenges, and Needs
Gao, J., Xie, C., & Tao, C. (2016) · 2016 IEEE Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2016.63 · Cited by 98
Bian, J., Lyu, T., Loiacono, A., Viramontes, T. M., Lipori, G., Guo, Y., Wu, Y., Prosperi, M., George, T. J., Harle, C. A., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa245 · Cited by 92
The Effect of Improving Annotation Quality on Object Detection Datasets: A Preliminary Study
Ma, J., Ushiku, Y., & Sagara, M. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00532 · Cited by 85
Big Data Quality Dimensions: A Systematic Literature Review
Ramasamy, A., & Chowdhury, S. (2020) · Journal of Information Systems and Technology Management · doi:10.4301/s1807-1775202017003 · Cited by 83
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82 · cited 1×
“Yet, one of the main causes of undesirable learned patterns lies in biased training data 55,56 .”
BackgroundA Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82 · cited 1×
“Yet, one of the main causes of undesirable learned patterns lies in biased training data 55,56 .”
BackgroundThe effects of data sampling with deep learning and highly imbalanced big data
Johnson, J. M., & Khoshgoftaar, T. M. (2020) · Information Systems Frontiers · doi:10.1007/s10796-020-10022-7 · Cited by 81
Detecting potential labeling errors in microarrays by data perturbation
Malossini, A., Blanzieri, E., & Ng, R. T. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl346 · Cited by 80
Data Quality Management in Pharmacovigilance
Lindquist, M. (2004) · Drug Safety · doi:10.2165/00002018-200427120-00003 · Cited by 77 · cited 1×
“This concerns aspects such as data integration 60 , information quality management 61 , ETL processes in data warehouses 62 , or tools for data warehouses 63,64 which do not affect the behavioural characteristics of AI systems.”
BackgroundEntropy-based redundancy analysis and information screening
Li, Y., Yang, J., & Wen, J. (2023) · Digital Communications and Networks · doi:10.1016/j.dcan.2021.12.001 · Cited by 77 · cited 1×
“Finally, the highly-discussed dimension target class balance pays tri-bute to the technical requirements of ML 140,141,144,150,159 .”
BackgroundProvenance documentation to enable explainable and trustworthy AI: A literature review
Kale, A., Nguyen, T., Harris, F. C., Li, C., Zhang, J., & Ma, X. (2023) · Data Intelligence · doi:10.1162/dint_a_00119 · Cited by 77
Non-Local Auto-Encoder With Collaborative Stabilization for Image Restoration
Wang, R., & Tao, D. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2541318 · Cited by 76 · cited 1×
“The overarching topics Theearly data quality research in the 1980s and 1990s uncovered the lackofobjectivemeasurestoassesdataquality,whichledtotheintroduction oftask-dependentdimensionsandtheestablishmentofadataqualityfra-meworkfromtheperspectiveofthedataconsumer 57 .”
BackgroundMachine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
Glocker, B., Robinson, R., Castro, D. C., Dou, Q., & Konukoglu, E. (2019) · arXiv · doi:10.48550/arxiv.1910.04597 · Cited by 74
Healthsheet: Development of a Transparency Artifact for Health Datasets
Rostamzadeh, N., Mincu, D., Roy, S., Smart, A., Wilcox, L., Pushkarna, M., Schrouff, J., Amironesei, R., Moorosi, N., & Heller, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533239 · Cited by 72 · cited 3×
“While some initiatives cover the entire ML system and development pipeline (e.g., MINIMAR 39 , FactSheets 40 ), others are concerned with documentation surrounding the model (e.g., Model Cards 41 ), and still others concentrate on the documentation of datasets (e.g., Datasheets 42 , STANDING Together 43,44 , Dataset N…”
BackgroundAccuracy and data efficiency in deep learning models of protein expression
Nikolados, E.-M., Wongprommoon, A., Aodha, O. M., Cambray, G., & Oyarzún, D. A. (2022) · Nature Communications · doi:10.1038/s41467-022-34902-5 · Cited by 70
Barragán-Montero, A. M., Thomas, M., Defraene, G., Michiels, S., Haustermans, K., Lee, J. A., & Sterpin, E. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.02.026 · Cited by 69
Effects of data quality and quantity on deep learning for protein-ligand binding affinity prediction
Fan, F. J., & Shi, Y. Y. (2022) · Bioorganic & Medicinal Chemistry · doi:10.1016/j.bmc.2022.117003 · Cited by 68
A Data Quality Ontology for the Secondary Use of EHR Data.
(2015) · PubMed Central · Cited by 66
Toward Sustainability: Trade-Off Between Data Quality and Quantity in Crop Pest Recognition
Li, Y., & Chao, X. (2021) · Frontiers in Plant Science · doi:10.3389/fpls.2021.811241 · Cited by 65
Schmarje, L., Grossmann, V., Zelenka, C., Dippel, S., Kiko, R., Oszust, M., Pastell, M., Stracke, J., Valros, A., Volkmann, N., et al. (2022) · neural information processing systems · doi:10.52202/068431-2407 · Cited by 64
Jenkinson, C., Fitzpatrick, R. M., Norquist, J. M., Findley, L. J. J., & Hughes, K. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00148-3 · Cited by 63 · cited 1×
“This represents the literature corpus that serves as a foundation for answering the research question.”
BackgroundRanjan, R., Sharrer, K. L., Tsukuda, S. M., & Good, C. M. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107644 · Cited by 62
Defining & assessing the quality, usability, and utilization of immunization data
Bloland, P., & MacNeil, A. (2019) · BMC Public Health · doi:10.1186/s12889-019-6709-1 · Cited by 59
Quality of input data in emergency department simulations: Framework and assessment techniques
Vanbrabant, L., Martin, N., Ramaekers, K. M., & Braekers, K. (2018) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2018.12.002 · Cited by 58
Wang, L., & Jackson, D. A. (2023) · Landscape Ecology · doi:10.1007/s10980-023-01771-2 · Cited by 58
Introduction: Measuring Data Quality for Ongoing Improvement
Sebastian-Coleman, L. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-397033-6.00020-1 · Cited by 57
Data Curation and Quality Evaluation for Machine Learning-Based Cyber Intrusion Detection
Tran, N., Chen, H., Bhuyan, J. N., & Ding, J. (2022) · IEEE Access · doi:10.1109/access.2022.3211313 · Cited by 54 · cited 1×
“…studies increasingly look at data quality in the context of sequential data (often time series) 133 – 139 , images 119, 139 – 182 , natural language 120, 121 , 145,182 – 186 or other complex types When it comes to the concrete behaviour change of the DL algorithm, most of the DL papers in the…”
Generalizable Multi-Site Training and Testing Of Deep Neural Networks Using Image Normalization
Onofrey, J. A., Casetti-Dinescu, D. I., Lauritzen, A. D., Sarkar, S., Venkataraman, R., Fan, R. E., Sonn, G. A., Sprenkle, P. C., Staib, L. H., & Papademetris, X. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759295 · Cited by 52
Imbalanced Data Problems in Deep Learning-Based Side-Channel Attacks: Analysis and Solution
Ito, A., Saito, K., Ueno, R., & Homma, N. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3092050 · Cited by 52
Impact of ECG Dataset Diversity on Generalization of CNN Model for Detecting QRS Complex
Habib, A., Karmakar, C. K., & Yearwood, J. L. (2019) · IEEE Access · doi:10.1109/access.2019.2927726 · Cited by 49
Does Image Anonymization Impact Computer Vision Training?
Hukkelås, H., & Lindseth, F. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00019 · Cited by 48
Declerck, J., Kalra, D., Stichele, R. V., & Coorevits, P. L. (2024) · JMIR Medical Informatics · doi:10.2196/51560 · Cited by 44 · cited 2×
“The various frameworks still suffered from inconsistent terminology and attempts were made to harmonise the de fi nitions and assessment 96 – 103 .”
BackgroundJuddoo, S., George, C., Duquenoy, P., & Windridge, D. (2018) · Applied System Innovation · doi:10.3390/asi1040043 · Cited by 43
Wook, M., Hasbullah, N. A., Zainudin, N. M., Jabar, Z. Z. A., Ramli, S., Razali, N. A. M., & Yusop, N. M. M. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00439-5 · Cited by 43
Data Quality for Security Challenges
Verma, R. M., Zeng, V., & Faridi, H. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3363267 · Cited by 41 · cited 1×
“For the dimension device error , we distinguish between the subdimension accuracy , the systematic deviation from the ground truth (also called bias), and the sub-dimension precision , the variance of the data around a mean value (also called noise).”
MethodXu, G., Yue, Q., Liu, X., & Chen, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121686 · Cited by 41
Impacts of Dirty Data on Classification and Clustering Models: An Experimental Evaluation
Qi, Z., Wang, H., & Wang, A. (2021) · Journal of Computer Science and Technology · doi:10.1007/s11390-021-1344-6 · Cited by 40
Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation
Li, Y., Zhao, C., & Caragea, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.511 · Cited by 40
From Theory to Practice: A Data Quality Framework for Classification Tasks
Corrales, D. C., Ledezma, A., & Corrales, J. C. (2018) · Symmetry · doi:10.3390/sym10070248 · Cited by 38
Systematic auditing is essential to debiasing machine learning in biology
Eid, F.-E., Elmarakeby, H. A., Chan, Y. A., Fornelos, N., ElHefnawi, M., Van Allen, E. M., Heath, L. S., & Lage, K. (2021) · Communications Biology · doi:10.1038/s42003-021-01674-5 · Cited by 38
Distributed deep learning across multisite datasets for generalized CT hemorrhage segmentation
Remedios, S. W., Roy, S., Bermudez, C., Patel, M. B., Butman, J. A., Landman, B. A., & Pham, D. L. (2020) · Medical Physics · doi:10.1002/mp.13880 · Cited by 37 · cited 1×
“Appropriately investigating such questions can provide a strong indication for the applicability and generalisability of the ML application in different environments 220 – 223 .”
BackgroundCao, W., Hu, L., Gao, J., Wang, X., & Ming, Z. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04719-8 · Cited by 36
(2022) · doi:10.1145/3491102.3501868 · Cited by 36
Identifying Mislabeled Instances in Classification Datasets
Muller, N. M., & Markert, K. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851920 · Cited by 34
Agnew, C., Eising, C., Denny, P. E., Scanlan, A., van de Ven, P. W. J., & Grua, E. M. (2023) · IEEE Access · doi:10.1109/access.2023.3256723 · Cited by 34
Hu, J., & Wang, J. (2023) · Engineering Geology · doi:10.1016/j.enggeo.2023.107254 · Cited by 34
Liaw, S.-T., Pearce, C., Liyanage, H., Cheah-Liaw, G. S., & De Lusignan, S. (2014) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v21i4.87 · Cited by 32 · cited 3×
“The initial search of the databases Web of Science, PubMed and ACM Digital Library was performed on the 12th of April 2024 and yielded 4633 unique results.”
BackgroundData Quality in Clinical Research
Nahm, M. (2012) · Health Informatics · doi:10.1007/978-1-84882-448-5_10 · Cited by 32
Data Quality in Health Care: Main Concepts and Assessment Methodologies
Mashoufi, M., Ayatollahi, H., Khorasani-Zavareh, D., & Talebi Azad Boni, T. (2023) · Methods of Information in Medicine · doi:10.1055/s-0043-1761500 · Cited by 32
Lessons from debiasing data for fair and accurate predictive modeling in education
Sha, L., Gašević, D., & Chen, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120323 · Cited by 31
A Review of the State of the Art of Data Quality in Healthcare
Liu, C., Talaei‐Khoei, A., Storey, V. C., & Peng, G. (2023) · Journal of Global Information Management · doi:10.4018/jgim.316236 · Cited by 31
Data Processing and Model Selection for Machine Learning-based Network Intrusion Detection
Sahu, A., Mao, Z., Davis, K., & Goulart, A. (2020) · IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR) · doi:10.1109/cqr47547.2020.9101394 · Cited by 29
Lee, J. H., & You, S. J. (2024) · IEEE Access · doi:10.1109/access.2024.3352146 · Cited by 29 · cited 1×
“We note that aspects such as anonymisation or pseudonymisation may impact the quality of the content of a dataset by, e.g., removing information 167 .”
BackgroundSebastião, Y. V., Metzger, G. A., Chisolm, D. J., Xiang, H., & Cooper, J. N. (2021) · Injury Epidemiology · doi:10.1186/s40621-021-00298-x · Cited by 28 · cited 1×
“The accepted terms were copied into an Excel sheet, which served asa starting templatefortheMETRIC-framework.”
MethodTaran, V., Gordienko, Y. G., Rokovyi, A., Alienin, O., & Stirenko, S. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-16621-2_17 · Cited by 25
Impact of annotation quality on model performance of welding defect detection using deep learning
Cui, J., Zhang, B., Wang, X., Wu, J., Liu, J., Li, Y., Zhi, X., Zhang, W., & Yu, X. (2024) · Welding in the World · doi:10.1007/s40194-024-01710-y · Cited by 25
Analyses on Influence of Training Data Set to Neural Network Supervised Learning Performance
Zhou, Y., & Wu, Y. (2011) · Advances in Intelligent and Soft Computing · doi:10.1007/978-3-642-23753-9_4 · Cited by 24
Managing Organizational Data Resources: Quality Dimensions
Yoon, V., Aiken, P. H., & Guimãrães, T. (2000) · Information Resources Management Journal · doi:10.4018/irmj.2000070101 · Cited by 24
From Data Quality to Model Quality: An Exploratory Study on Deep Learning
He, T., Yu, S., Wang, Z., Li, J., & Chen, Z. (2019) · Proceedings of the 11th Asia-Pacific Symposium on Internetware · doi:10.1145/3361242.3361260 · Cited by 24
A study of real-world micrograph data quality and machine learning model robustness
Zhong, X., Gallagher, B. J., Eves, K. L., Robertson, E. L., Mundhenk, T. N., & Han, T. Y.-J. (2021) · npj Computational Materials · doi:10.1038/s41524-021-00616-3 · Cited by 24
Training dataset reduction on generative adversarial network
Nuha, F. U., & Afiahayati (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.10.513 · Cited by 21
Barakat, M. S., Field, M., Ghose, A., Stirling, D., Holloway, L., Vinod, S., Dekker, A., & Thwaites, D. (2017) · Health Information Science and Systems · doi:10.1007/s13755-017-0039-4 · Cited by 20 · cited 1×
“This de fi nition has been picked up by other researchers, as well 100,106,126 .”
BackgroundTahar, K., Martin, T., Mou, Y., Verbuecheln, R., Graeßner, H., & Krefting, D. (2023) · Methods of Information in Medicine · doi:10.1055/a-2006-1018 · Cited by 19 · cited 1×
“Thus, data quality frameworks for particular data types such as immunisation data, public health data, multi-centre healthcare data or similar were put forward 88 – 95 .”
BackgroundGüneş, A. M., van Rooij, W., Gulshad, S., Slotman, B. J., Dahele, M. R., & Verbakel, W. F. A. R. (2023) · Medical Physics · doi:10.1002/mp.16437 · Cited by 18
How false data affects machine learning models in electrochemistry?
Deshsorn, K., Lawtrakul, L., & Iamprasertkun, P. (2024) · Journal of Power Sources · doi:10.1016/j.jpowsour.2024.234127 · Cited by 18 · cited 1×
“Many records investigate tabular data while utilising both simpler and more advanced architectures 114 – 132 .”
MethodImpact of noise on credit risk prediction: Does data quality really matter?
Twala, B. (2013) · Intelligent Data Analysis · doi:10.3233/ida-130623 · Cited by 17
Understanding the Challenges When 3D Semantic Segmentation Faces Class Imbalanced and OOD Data
Pan, Y., Xie, F., & Zhao, H. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3256442 · Cited by 17
Narrowing the Gap: Improved Detector Training With Noisy Location Annotations
Wang, S., Gao, J. G., Li, B., & Hu, W. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3211468 · Cited by 15
The uncovered biases and errors in clinical determination of bone age by using deep learning models
Bai, M., Gao, L., Ji, M., Ge, J., Huang, L., Qiao, H., Xiao, J. Z., Chen, X., Yang, B., Sun, Y., et al. (2022) · European Radiology · doi:10.1007/s00330-022-09330-0 · Cited by 14
Data Quality Matters: A Case Study of Obsolete Comment Detection
Xu, S., Yao, Y., Feng, X., Gu, T., Xu, J., & Ma, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00074 · Cited by 14
Investigating the Impact of Image Quality on Endoscopic AI Model Performance
Jaspers, T. J. M., Boers, T. G. W., Kusters, C. H. J., Jong, M. R., Jukema, J. B., de Groof, A. J., Bergman, J. J. G. H. M., de With, P. H. N., & van der Sommen, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47076-9_4 · Cited by 14
Exploring Strategies for Training Deep Neural Networks
LarochelleHugo, BengioYoshua, LouradourJérôme, et al. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577070 · Cited by 13
A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings
Benedick, P.-L., Robert, J., & Le Traon, Y. (2021) · Sensors · doi:10.3390/s21186195 · Cited by 12
CONCEPTUALIZING BIG DATA QUALITY FRAMEWORK FROM A SYSTEMATIC LITERATURE REVIEW PERSPECTIVE
Ijab, M. T., Salwana, E., & Nayan, N. M. (2019) · Malaysian Journal of Computer Science · doi:10.22452/mjcs.sp2019no1.2 · Cited by 12 · cited 1×
“This development is re fl ected by a small wave of papers published between 2015 and 2020 106 – 113 .”
BackgroundMulti-Center Healthcare Data Quality Measurement Model and Assessment Using OMOP CDM
Kim, K., Choi, W., Ko, S., Chang, D. J., Chung, Y. W., Chang, S., Kim, J. K., Kim, D., & Choi, I. (2021) · Applied Sciences · doi:10.3390/app11199188 · Cited by 12
Annotation quality vs. quantity for deep-learned medical image segmentation
Wesemeyer, T., Jauer, M.-L., & Deserno, T. M. (2021) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.2582226 · Cited by 11
Schmarje, L., Grossmann, V., Zelenka, C., et al. (2022) · Helmholtz Centre for Ocean Research Kiel (GEOMAR) · doi:10.48550/arxiv.2207.06214 · Cited by 11
Studying the Effects of Training Data on Machine Learning-Based Procedural Content Generation
Snodgrass, S., Summerville, A., & Ontañón, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v13i1.12930 · Cited by 10
Impact of Anonymization on Vehicle Detector Performance
Schnabel, L., Matzka, S., Stellmacher, M., Pätzold, M., & Matthes, E. (2019) · International Conference on Artificial Intelligence for Industries (AI4I) · doi:10.1109/ai4i46381.2019.00016 · Cited by 9
Learn to Train: Improving Training Data for a Neural Network to Detect Pecking Injuries in Turkeys
Volkmann, N., Brünger, J., Stracke, J., Zelenka, C., Koch, R., Kemper, N., & Spindler, B. (2021) · Animals · doi:10.3390/ani11092655 · Cited by 9
Loshin, D. S. (2010) · Elsevier eBooks · doi:10.1016/b978-0-12-373717-5.00008-7 · Cited by 8
Data Quality for Medical Data Lakelands
Eder, J., & Shekhovtsov, V. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63924-2_2 · Cited by 8
Training data composition affects performance of protein structure analysis algorithms.
Derry, A., Carpenter, K. A., & Altman, R. B. (2022) · PubMed · Cited by 8
The Impact of Data Quality on Software Testing Effort Prediction
Radliński, Ł. (2023) · Electronics · doi:10.3390/electronics12071656 · Cited by 8
Improving Efficiency in Facial Recognition Tasks Through a Dataset Optimization Approach
Vilaça, L., Viana, P., Carvalho, P., & Andrade, M. T. (2024) · IEEE Access · doi:10.1109/access.2024.3371020 · Cited by 8
A Data-Centric Approach for Training Deep Neural Networks with Less DataPreprint
Motamedi, M., Sakharnykh, N., & Kaldewey, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03613 · Cited by 8
Jaramillo, D. (2022) · Radiology · doi:10.1148/radiol.211891 · Cited by 7
Tools for data warehouse quality
Gebhardt, M., Jarke, M., Jeusfeld, M. A., Quix, C., & Sklorz, S. (2002) · International Conference on Scientific and Statistical Database Management · doi:10.1109/ssdm.1998.688130 · Cited by 6 · cited 1×
“After title and abstract screening, adding references of the remaining records ( ‘ snowballing ’ ) and full text assessment, we fi nd 120 records that match our eligibility criteria (see Methods).”
MethodLake, S. E., & Tsai, C. (2022) · Astronomy and Computing · doi:10.1016/j.ascom.2022.100617 · Cited by 6
Enhancing Pest Detection Models Through Improved Annotations
Costa, D., Silva, C., Cósta, J., & Ribeiro, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49011-8_29 · Cited by 6
The STANDING Together collaboration (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10048355 · Cited by 6
On Studying the Effect of Data Quality on Classification Performances
Jouseau, R., Salva, S., & Samir, C. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-21753-1_9 · Cited by 5
Research works printed but not linked: 8
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T. & Dwork, C. Learning fair representations. In Proc. 30th International Conference on Machine Learning, vol. 28, 325–333 (PMLR, Atlanta, Georgia, USA, 2013).
- Long, J., Richards, J. & Seko, C. The Canadian Institute for Health Information Data Quality Framework, version 1: a meta-evaluation and future directions. In Proc. Sixth International Conference on Information Quality, 370–383 (2001).
- Almutiry, O., Wills, G., Alwabel, A., Crowder, R. & Walters, R. Toward a framework for data quality in cloud-based health information system. In Proc. International Conference on Information Society (i-Society 2013), 153–157 (IEEE, 2013).
- Gudivada, V., Apon, A. & Ding, J. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. Int. J. Adv. Softw. 10, 1–20 (2017).
- Zhang, H., Singh, H., Ghassemi, M. & Joshi, S. ‘Why did the model fail?’ Attributing model performance changes to distribution shifts. In Proc. 40th International Conference on Machine Learning, Vol. 202, 41550–41578 (2023).
- Hong, S. & Shen, J. Impact of training size on deep learning performance in in vivo 1H MRS. In Proc.ISMRM & SMRT Annual Meeting & Exhibition (2021).
- Thaler, R. H. & Sunstein, C. R. Nudge: Improving Decisions About Health, Wealth, and Happiness (Yale University Press, 2009).
- He, H., Bai, Y., Garcia, E. A. & Li, S. Adasyn: adaptive synthetic sampling approach for imbalanced learning. In Proc. IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) (IEEE, Hong Kong, 2008).
Cited by
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 61
“…development, including dataset sources, size, geography, population demographics, disease prevalence, training – test split strategy, evaluation metrics, external validation sites (if any), subgroup results (if any), planned update pathways, and relevant model architecture information 42,43 .”
Gebler, R., Reinecke, I., Sedlmayr, M., & Goldammer, M. (2025) · Journal of Medical Internet Research · doi:10.2196/74976 · Cited by 37
“…out” and provide reliable outputs [5] • Transparency through provenance, versioning, and integrity to ensure reproducible and auditable results [5] • Standards-based interoperability to ensure data content and exchange [6] • Ability to handle large sized, multimodal datasets [5] These…”
InfluentialYang, L., Allen, G., Zhang, Z., & Zhao, Y. (2024) · Buildings · doi:10.3390/buildings15010021 · Cited by 35
A review of recent artificial intelligence for traditional medicine
Hou, C., Gao, Y., Lin, X., Wu, J., Li, N., Lv, H., & Chu, W. C. (2025) · Journal of Traditional and Complementary Medicine · doi:10.1016/j.jtcme.2025.02.009 · Cited by 29
Dwyer, B., Flathers, M., Sano, A., Dempsey, A. G., Cipriani, A., Gazi, A. H., Hill, B., Gorban, C., Rodríguez, C. I., Stromeyer, C., et al. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00049-6 · Cited by 39
Metadata
- Authors
- Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T.
- Affiliations
- Physikalisch-Technische Bundesanstalt (DE)
- Corresponding author
- Schwabe, Daniel
- Venue
- npj Digital Medicine
- Published
- 2024-08-03
- Volume
- 7
- Issue
- 1
- Pages
- 203-203
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11297942
- OpenAlex
- W4401283978
- Cited by
- 169
- References
- 194
- Citations by year
- 2026: 92 · 2025: 69 · 2024: 8
- FWCI
- 3.85
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission (101100700)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- data quality assessment, trustworthy AI, machine learning in medicine, deep learning, training data bias, robustness, interpretability, reference dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.