Potential limitations in COVID-19 machine learning due to data source variability: A case study in the nCov2019 dataset
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Objective: The lack of representative coronavirus disease 2019 (COVID-19) data is a bottleneck for reliable and generalizable machine learning. Data sharing is insufficient without data quality, in which source variability plays an important role. We showcase and discuss potential biases from data source variability for COVID-19 machine learning. Materials and Methods: We used the publicly available nCov2019 dataset, including patient-level data from several countries. We aimed to the discovery and classification of severity subgroups using symptoms and comorbidities. Results: Cases from the 2 countries with the highest prevalence were divided into separate subgroups with distinct severity manifestations. This variability can reduce the representativeness of training data with respect the model target populations and increase model complexity at risk of overfitting. Conclusions: Data source variability is a potential contributor to bias in distributed research networks. We call for systematic assessment and reporting of data source variability and data quality in COVID-19 data sharing, as key information for reliable and generalizable machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · cited 1×
“9,10 In the presence of multisource variability, we could argue for building local predictive models for each data source or, otherwise, building a global model including all the data sources.”
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 2×
“Gianfrancesco et al 13 notably remarked that “algorithms may not offer benefit to people whose data are missing from the data set.”
BackgroundUnintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa196 · Cited by 553 · cited 1×
“15,27,28 The ideal goal is to provide high-quality, homogeneous data, but this is not always straightforward.”
BackgroundEpidemiological data from the COVID-19 outbreak, real-time case information
Xu, B., Gutierrez, B., Mekaru, S., Sewalk, K., Goodwin, L., Loskill, A., Cohn, E. L., Hswen, Y., Hill, S. C., Cobo, M. M., et al. (2020) · Scientific Data · doi:10.1038/s41597-020-0448-0 · Cited by 396 · cited 1×
“Philippines would be assigned only to the high-severity subgroups 2, 3, 5, and 6.”
BackgroundRandomized clinical trials of artificial intelligence
Angus, D. C. (2020) · JAMA · doi:10.1001/jama.2020.1039 · Cited by 112
Cited by
Data drift in medical machine learning: implications and potential remedies
Sahiner, B., Chen, W., Samala, R. K., & Petrick, N. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220878 · Cited by 279
Bias in algorithms of AI systems developed for COVID-19: A scoping review
Delgado, J., de Manuel, A., Parra, I., Moyano, C., Rueda, J., Guersenzvaig, A., Ausin, T., Cruz, M., Casacuberta, D., & Puyol, A. (2022) · Journal of Bioethical Inquiry · doi:10.1007/s11673-022-10200-z · Cited by 70
“For countries, we found five articles from the United States (Grantz et al. 2020; Marabelli, Vaast, and Li 2021; Moss and Metcalf 2020; Röösli, Rice, and Hernandez-Boussard 2021; Shachar, Gerke, and Adashi 2020 ), one from the United States and South-Korea (Park, Choi, and Ko 2020), two from Australia (Scott and Coier…”
BackgroundInfluentialSáez, C., Ferri, P., & García-Gómez, J. M. (2024) · Journal of Medical Internet Research · doi:10.2196/50295 · Cited by 36
Self-citationNguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., & Zomaya, A. Y. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3120998 · Cited by 5
“However, in practical scenarios, COVID-19 image datasets are distributed across multiple sites, where data in each site is too limited in quantity and diversity to train accurately a GAN for the entire data population [10].”
Background
Metadata
- Authors
- Sáez, C., Romero, N., Conejero, J. A., & García-Gómez, J. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-10-07
- Volume
- 28
- Issue
- 2
- Pages
- 360-364
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3092635280
- Cited by
- 77
- References
- 7
- FWCI
- 5.11
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.