Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Objective: In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations. Materials and Methods: We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements. Results: Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback. Discussion: We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate. Conclusion: By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542 · cited 1×
“14 In a federated data network, each participating site’s data stay behind its institutional firewall, but are structured according to a CDM.”
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 2×
“17,19 22 DQ evaluations must take into account the initial purpose of the data in their source systems, as well as the intended use of these data once harmonization processes have transformed them for use in secondary research.”
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 697 · cited 2×
“20,29 By participating in a consortium like N3C, sites receive routine feedback on their overall quality with tactical information on ways to address local issues.”
Self-citationThe National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa196 · Cited by 553
Self-citationStang, P. E., Ryan, P. B., Racoosin, J. A., Overhage, J. M., Hartzema, A. G., Reich, C., Welebob, E., Scarnecchia, T., & Woodcock, J. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-9-201011020-00010 · Cited by 441
Defining and measuring completeness of electronic health records for secondary use
Weiskopf, N. G., Hripcsak, G., Swaminathan, S., & Weng, C. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.010 · Cited by 367 · cited 2×
“Data quality DQ emerged in the 2010s as an explicit subject of attention and research in informatics.(22,28,29) Notably, the Patient-Centered Outcomes Research Institute (PCORI) funded an effort to define DQ, resulting in the “Harmonization” framework of Kahn and colleagues.”
Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012) · Medical Care · doi:10.1097/mlr.0b013e318257dd67 · Cited by 260 · cited 1×
“18 Despite their importance, data quality (DQ) checks are inconsistently applied and implementation methodologies are largely not evaluated.”
Self-citationQualls, L. G., Phillips, T. A., Hammill, B. G., Topping, J., Louzao, D. M., Brown, J. S., Curtis, L. H., & Marsolo, K. (2018) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.199 · Cited by 116
Bian, J., Lyu, T., Loiacono, A., Viramontes, T. M., Lipori, G., Guo, Y., Wu, Y., Prosperi, M., George, T. J., Harle, C. A., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa245 · Cited by 92
Johnson, K. E., Kamineni, A., Fuller, S., Olmstead, D., & Wernli, K. J. (2014) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1058 · Cited by 39 · cited 1×
“19,23,24 Each of the aforementioned federated networks has methods to promote local DQ and adherence to data model conventions; these methods vary in maturity.”
Cited by
Electronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171
Federated learning for healthcare applications
Chaddad, A., Wu, Y., & Desrosiers, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3325822 · Cited by 136
“Poor data quality, small data set size, and incomplete labeling can prevent model training, especially in medical domains where acquiring labeled data is challenging [191].”
BackgroundAtalla, A. D. G., El-Gawad Mousa, M. A., Hashish, E. A. A., Elseesy, N. A. M., Abd El kader Mohamed, A. I., & Sobhi Mohamed, S. M. (2025) · BMC Nursing · doi:10.1186/s12912-025-03306-3 · Cited by 58
Data Quality–Driven Improvement in Health Care: Systematic Literature Review
Lighterness, A., Adcock, M., Scanlon, L. A., & Price, G. (2024) · Journal of Medical Internet Research · doi:10.2196/57615 · Cited by 26
“For uniqueness, studies assessed if records were unexpectedly duplicated, for example, in primary keys [79] or patient names [95].”
Influential
Metadata
- Authors
- Pfaff, E. R., Girvin, A. T., Gabriel, D. L., Kostka, K., Morris, M., Palchuk, M. B., Lehmann, H. P., Amor, B., Bissell, M., Bradwell, K. R., Gold, S., Hong, S. S., Loomba, J., Manna, A., McMurry, J. A., Niehaus, E., Qureshi, N., Walden, A., Zhang, X. T., Zhu, R. L., Moffitt, R. A., Haendel, M. A., Chute, C. G., The N3C Consortium, Adams, W. G., Al-Shukri, S., Anzalone, A., Baghal, A., Bennett, T. D., Bernstam, E. V., Bernstam, E. V., Bissell, M. M., Bush, B., Campion, T. R., Castro, V., Chang, J., Chaudhari, D. D., Chen, W., Chu, S., Cimino, J. J., Crandall, K. A., Crooks, M., Davies, S. J. D., DiPalazzo, J., Dorr, D., Eckrich, D., Eltinge, S. E., Fort, D. G., Golovko, G., Gupta, S., Haendel, M. A., Hajagos, J. G., Hanauer, D. A., Harnett, B. M., Horswell, R., Huang, N., Johnson, S. G., Kahn, M., Khanipov, K., Kieler, C., Luzuriaga, K. R. D., Maidlow, S., Martinez, A., Mathew, J., McClay, J. C., McMahan, G., Melancon, B., Meystre, S., Miele, L., Morizono, H., Pablo, R., Patel, L., Phuong, J., Popham, D. J., Pulgarin, C., Santos, C., Sarkar, I. N., Sazo, N., Setoguchi, S., Soby, S., Surampalli, S., Suver, C., Vangala, U. M. R., Visweswaran, S., Oehsen, J. V., Walters, K. M., Wiley, L., Williams, D. A., & Zai, A.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-11-02
- Volume
- 29
- Issue
- 4
- Pages
- 609-618
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3203418161
- Cited by
- 73
- References
- 11
- FWCI
- 8.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative. https://app.cochat.ai/papers/10.1093/jamia/ocab217
- The paper
- Pfaff, E. R., Girvin, A. T., Gabriel, D. L., Kostka, K., Morris, M., Palchuk, M. B., Lehmann, H. P., Amor, B., Bissell, M., Bradwell, K. R., Gold, S., Hong, S. S., Loomba, J., Manna, A., McMurry, J. A., Niehaus, E., Qureshi, N., Walden, A., Zhang, X. T., . . . Zai, A. (2022). Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocab217
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.