CoChat
Sign Up
Cited by 553Research Data Management PracticesElectronic Health Records SystemsCOVID-19 and healthcare impacts

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment

Open access
Haendel, Melissa A, Chute, Christopher G, Bennett, Tellen D, Eichmann, David A, Guinney, Justin, Kibbe, Warren A, Payne, Philip R O, Pfaff, Emily R +49 more·2021·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocaa196PMC PMC7454687
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers. Materials and Methods: The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics. Results: Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access. Conclusions: The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa196

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 67

  1. [2]Kissler SM, Tedijanto C, Goldstein E, et al Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period. Science 2020; 368 (6493): 860– 8. 32291278 PMC7164482
  2. [4]Visweswaran S, Becich MJ, D’Itri VS, et al Accrual to Clinical Trials (ACT): A Clinical and Translational Science Award Consortium Network. JAMIA Open 2018; 1 (2): 147– 52. 30474072 PMC6241502
  3. [5]Fleurence RL, Curtis LH, Califf RM, et al Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014; 21 (4): 578– 82. 24821743 PMC4078292
  4. [7]Findlay S. The FDA’s Sentinel Initiative. Health Policy Brief. Health Affairs 2015. https://www.healthaffairs.org/do/10.1377/hpb20150604.936915/full/healthpolicybrief_139.pdf Accessed June 7, 2020. · DOI
  5. [8]Topaloglu U, Palchuk MB. Using a federated network of real-world data to optimize clinical trials operations. JCO Clin Cancer Inform 2018; 2 (2): 1– 10. PMC6816049 30652541
  6. [9]Brat GA, Weber GM, Gehlenborg N, et al International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium. npj Digit Med 2020; 3: 109. PMC7438496 32864472
  7. [11]Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019; 380 (14): 1347– 58. 30943338
  8. [12]Yu K-H, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng 2018; 2 (10): 719– 31. 31015651
  9. [13]Kramer WG, Perentesis G, Affrime MB, et al Pharmacokinetics of dilevalol in normotensive and hypertensive volunteers. Am J Cardiol 1989; 63 (19): 7I– 11I. 2729127
  10. [14]Obermeyer Z, Emanuel EJ. Predicting the future—big data, machine learning, and clinical medicine. N Engl J Med 2016; 375 (13): 1216– 9. 27682033 PMC5070532
  11. [15]Wang Y, Zhao Y, Therneau TM, et al Unsupervised machine learning for the discovery of latent disease clusters and patient subgroups using electronic health records. J Biomed Inform 2020; 102: 103364. 31891765 PMC7028517
  12. [16]Li T, Sahu AK, Talwalkar A, et al Federated learning: challenges, methods, and future directions. IEEE Signal Process Mag 2020; 37 (3): 50– 60.
  13. [17]Zerka F, Barakat S, Walsh S, et al Systematic review of privacy-preserving distributed machine learning from federated databases in health care. JCO Clin Cancer Inform 2020; 4 (4): 184– 200. 32134684 PMC7113079
  14. [18]Liu P, Qi J, et al Federated machine learning: concept and applications. ACM Trans Intell Syst Technol 2019; 10 (6): 1– 19.
  15. [20]Mehra MR, Desai SS, Ruschitzka F, Patel AN. Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet 2020 May 22; doi: 10.1016/S0140-6736(20)31180-6. PMC7255293 32450107 · DOI
  16. [21]Mehra MR, Desai SS, Kuy S, et al Retraction: cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med 2020; 382 (25): e102. 32356626 PMC7206931
  17. [29]Grayson S, Suver C, Wilbanks J, et al Open Data Sharing in the 21st Century: Sage Bionetworks’ Qualified Research Program and Its Application in mHealth Data Release. SSRN 2019 Jan 19 [E-pub ahead of print]. doi: 10.2139/ssrn.3502410. · DOI
  18. [35]Raab GM, Nowok B, Dibben C. Guidelines for Producing Useful Synthetic Data. arXiv: 1712.04078; 2017. · arXiv
  19. [36]Snoke J, Raab GM, Nowok B, et al General and specific utility measures for synthetic data. J R Stat Soc A 2018; 181 (3): 663– 88.
  20. [41]Sprague ER. ORCID. J Med Libr Assoc 2017; 105: 207.
  21. [44]Katz DS, Smith AM. Transitive credit and JSON-LD J Open Res Soft 2015; 3: 14.
  22. [48]Burn E, You SC, Sena AG, et al An international characterisation of patients hospitalised with COVID-19 and a comparison with those previously hospitalised with influenza. medRxiv 2020. doi: 10.1101/2020.04.22.20074336. · DOI
  23. [51]Weber GM, Murphy SN, McMurry AJ, et al The Shared Health Research Information Network (SHRINE): a prototype federated query tool for clinical data repositories. J Am Med Inform Assoc 2009; 16 (5): 624– 30. 19567788 PMC2744712
  24. [58]Chute CG. Clinical classification and terminology: some history and current observations. J Am Med Inform Assoc 2000; 7 (3): 298– 303. 10833167 PMC61433
  25. [59]Haendel MA, Chute CG, Robinson PN. Classification, ontology, and precision medicine. N Engl J Med 2018; 379 (15): 1452– 62. 30304648 PMC6503847
  26. [61]Chute CG, Huff SM. The pluripotent rendering of clinical data for precision medicine. Stud Health Technol Inform 2017; 245: 337– 40. 29295111
  27. [65]Kahn MG, Batson D, Schilling LM. Data model considerations for clinical effectiveness researchers. Med Care 2012; 50: S60– 7. 22692260 PMC3824370
  28. [66]Ogunyemi OI, Meeker D, Kim H-E, et al Identifying appropriate reference data models for comparative effectiveness research (CER) studies based on data from clinical information systems. Med Care 2013; 51 (8 Suppl 3): S45– 52. 23774519
  29. [71]Kahn MG, Brown JS, Chun AT, et al Transparent reporting of data quality in distributed data networks. EGEMS (Wash DC) 2015; 3 (1): 1052. 25992385 PMC4434997
  30. [72]Khare R, Utidjian L, Ruth BJ, et al A longitudinal analysis of data quality in a large pediatric data research network. J Am Med Inform Assoc 2017; 24 (6): 1072– 9. 28398525 PMC6259665
  31. [73]Weiskopf NG, Hripcsak G, Swaminathan S, et al Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46 (5): 830– 6. 23820016 PMC3810243
  32. [75]Zozus M. The Data Book: Collection and Management of Research Data. Boca Raton, FL: CRC Press; 2017.
  33. [76]Kahn MG, Eliason BB, Bathurst J. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proc 2010; 2010: 356– 60. 21347000 PMC3041459
  34. [84]Eagleton MJ, Kashyap VS. Introduction. J Vasc Surg 2020; 72 (1): e4– 5. 32553410 PMC7295701
  35. [85]Dong X, Li J, Soysal E. COVID-19 TestNorm—a tool to normalize COVID-19 testing names to LOINC codes. J Am Med Inform Assoc 2020; 27 (9): 1437– 42. 32569358 PMC7337837
  36. [86]Lane J, Schur C. Balancing access to health data and privacy: a review of the issues and approaches for the future. Health Serv Res 2010; 45 (5p2): 1456– 67. 21054366 PMC2965886
  37. [87]Hripcsak G, Shang N, Peissig PL, et al Facilitating phenotype transfer using a common data model. J Biomed Inform 2019; 96: 103253. 31325501 PMC6697565
  38. [88]Swerdel JN, Hripcsak G, Ryan PB. PheValuator: development and evaluation of a phenotype algorithm evaluator. J Biomed Inform 2019; 97: 103258. 31369862 PMC7736922
  39. [89]Reps JM, Schuemie MJ, Suchard MA, et al Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc 2018; 25 (8): 969– 75. 29718407 PMC6077830
  40. [90]Schuemie MJ, Cepede MS, Suchard MA, et al How confident are we about observational findings in health care: a benchmark study. Harv Data Sci Rev 2020; 2 (1); doi: 10.1162/99608f92.147cc28e PMC7755157 33367288 · DOI
  41. [92]Schuemie MJ, Hripcsak G, Ryan PB, et al Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data. Proc Natl Acad Sci U S A 2018; 115 (11): 2571– 7. 29531023 PMC5856503
  42. [93]Zhang XA, Yates A, Vasilevsky N, et al Semantic integration of clinical laboratory tests from electronic health records for deep phenotyping and biomarker discovery. NPJ Digit Med 2019; 2: 32. 31119199 PMC6527418
  43. [94]Biomedical Data Translator Consortium. Toward a universal biomedical data translator. Clin Transl Sci 2019; 12: 86– 90. 30412337 PMC6440568
  44. [95]Biomedical Data Translator Consortium. The biomedical data translator program: conception, culture, and community. Clin Transl Sci 2019; 12(2): 91– 4. 30412340 PMC6440573
  45. [96]Austin CP, Colvis CM, Southall NT. Deconstructing the translational tower of babel. Clin Transl Sci 2019; 12 (2): 85. 30412342 PMC6440561
  46. [99]Dobbins NJ, Spital CH, Black RA, et al Leaf: an open-source, model-agnostic, data-driven web application for cohort discovery and translational biomedical research. J Am Med Inform Assoc 2020; 27 (1): 109– 18. 31592524 PMC6913227
  47. [101]Brito JJ, Li J, Moore JH, et al Recommendations to enhance rigor and reproducibility in biomedical research. GigaScience 2020; 9 (6): giaa056 PMC7263079 32479592
  48. [102]Walonoski J, Kramer M, Nichols J, et al Synthea: an approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. J Am Med Inform Assoc 2018; 25 (3): 230– 8. 29025144 PMC7651916
  49. [103]Baowaly MK, Lin C-C, Liu C-L, et al Synthesizing electronic health records using improved generative adversarial networks. J Am Med Inform Assoc 2019; 26 (3): 228– 41. 30535151 PMC7647178
  50. [104]Chen J, Chun D, Patel M, et al The validity of synthetic clinical data: a validation study of a leading synthetic data generator (Synthea) using clinical quality measures. BMC Med Inform Decis Mak 2019; 19 (1): 44. 30871520 PMC6416981
  51. [105]Hayes J, Melis L, Danezis G, et al LOGAN: membership inference attacks against generative models. arXiv: 1705.07663; 2017. · arXiv
  52. [107]Foraker R, Mann DL, Payne PRO. Are synthetic data derivatives the future of translational medicine? J Am Coll Cardio Basic Trans Sci 2018; 3 (5): 716– 8. PMC6234614 30456342
  53. [109]Shickel B, Tighe PJ, Bihorac A, et al Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform 2018; 22 (5): 1589– 604. 29989977 PMC6043423
  54. [110]Luo Y, Wang F, Szolovits P. Tensor factorization toward precision medicine. Brief Bioinform 2017; 18 (3): 511– 4. 26994614 PMC6078180
  55. [111]Thompson AE, Ranard BL, Wei Y, et al Prone positioning in awake, nonintubated patients with COVID-19 hypoxemic respiratory failure. JAMA Intern Med 2020 Jun 17 [E-pub ahead of print]; doi: 10.1001/jamainternmed.2020.3030. PMC7301298 32584946 · DOI
  56. [112]Mehta P, McAuley DF, Brown M, et al COVID-19: consider cytokine storm syndromes and immunosuppression. Lancet 2020; 395 (10229): 1033– 4. 32192578 PMC7270045
  57. [113]Suo Q, Ma F, Yuan Y, et al Deep patient similarity learning for personalized healthcare. IEEE Trans Nanobiosci 2018; 17 (3): 219– 27. 29994534
  58. [114]Belhadjer Z, Méot M, Bajolle F, et al Acute heart failure in multisystem inflammatory syndrome in children (MIS-C) in the context of global SARS-CoV-2 pandemic. Circulation 2020; 142: 429– 36. doi: 10.1161/CIRCULATIONAHA.120.048360. 32418446 · DOI
  59. [115]Lin KJ, Rosenthal GE, Murphy SN, et al External validation of an algorithm to identify patients with high data-completeness in electronic health records for comparative effectiveness research. Clin Epidemiol 2020; 12: 133– 41. 32099479 PMC7007793
  60. [116]Kharrazi H, Lasser EC, Yasnoff WA, et al A proposed national research and development agenda for population health informatics: summary recommendations from a national expert workshop. J Am Med Inform Assoc 2017; 24 (1): 2– 12. 27018264 PMC5201177
  61. [117]Kharrazi H, Chi W, Chang H-Y, et al Comparing population-based risk-stratification model performance using demographic, diagnosis and medication data extracted from outpatient electronic health records versus administrative claims. Med Care 2017; 55 (8): 789– 96. 28598890
  62. [118]Williams DR, Cooper LA. COVID-19 and health equity-a new kind of ‘herd immunity’. JAMA 2020; 323 (24): 2478. 32391852
  63. [119]Glover RE, van Schalkwyk MC, Akl EA, et al A framework for identifying and mitigating the equity harms of COVID-19 policy interventions. J Clin Epidemiol 2020; 128: 35– 48. 32526461 PMC7280094
  64. [121]Millett GA, Jones AT, Benkeser D, et al Assessing differential impacts of COVID-19 on Black communities. Ann Epidemiol 2020; 47: 37– 44. 32419766 PMC7224670
  65. [122]Gamache R, Kharrazi H, Weiner JP. Public and population health informatics: the bridging of big data to benefit communities. Yearb Med Inform 2018; 27 (1): 199– 206. 30157524 PMC6115205
  66. [124]Cimino JJ, Ayres EJ, Remennik L, et al The National Institutes of Health’s Biomedical Translational Research Information System (BTRIS): design, contents, functionality and experience to date. J Biomed Inform 2014; 52: 11– 27. 24262893 PMC4026354
  67. [126]Hersh W, Cimino J, Payne PRO, et al Recommendations for the use of operational electronic health record data in comparative effectiveness research. EGEMS (Wash DC) 2013; 1 (1): 1018. 25848563 PMC4371471

Cited by

Sorted by citations

Authors
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., Spratt, H., Suver, C., Wilbanks, J., Wilcox, A. B., Williams, A. E., Wu, C., Blacketer, C., Bradford, R. L., Cimino, J. J., Clark, M., Colmenares, E. W., Francis, P. A., Gabriel, D., Graves, A., Hemadri, R., Hong, S. S., Hripscak, G., Jiao, D., Klann, J. G., Kostka, K., Lee, A. M., Lehmann, H. P., Lingrey, L., Miller, R. T., Morris, M., Murphy, S. N., Natarajan, K., Palchuk, M. B., Sheikh, U., Solbrig, H., Visweswaran, S., Walden, A., Walters, K. M., Weber, G. M., Zhang, X. T., Zhu, R. L., Amor, B., Girvin, A. T., Manna, A., Qureshi, N., Kurilla, M. G., Michael, S. G., Portilla, L. M., Rutter, J. L., Austin, C. P., Gersing, K. R., & the N3C Consortium
Affiliations
  • Oregon State University (US)
  • Johns Hopkins University (US)
  • University of Colorado Anschutz (US)
  • University of Iowa (US)
  • Sage Bionetworks (US)
  • Duke University (US)
  • Washington University in St. Louis (US)
  • University of North Carolina at Chapel Hill (US)
  • Jackson Laboratory (US)
  • Stony Brook University (US)
  • The University of Texas Medical Branch at Galveston (US)
  • University of Washington (US)
  • Tufts Medical Center (US)
  • Scripps Research Institute (US)
  • Janssen Research & Development (United States) (US)
  • University of Alabama at Birmingham (US)
  • National Center for Advancing Translational Sciences (US)
  • Columbia University (US)
  • Harvard University (US)
  • IQVIA (United States) (US)
  • TriNetX (United States) (US)
  • Tufts University (US)
  • University of Pittsburgh (US)
  • Mass General Brigham (US)
  • Columbia University Irving Medical Center (US)
  • Palantir Technologies, Palo Alto, California, USA
  • National Institutes of Health (US)
  • Office of the Director (US)
  • National Center for Advancing Translational Science, Bethesda, Maryland, USA
Corresponding author
  • Haendel, Melissa A
  • Chute, Christopher G
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2020-08-17
Volume
28
Issue
3
Pages
427-443
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocaa196
PMC
PMC7454687
OpenAlex
W3049452588
Cited by
553
References
10
Citations by year
2026: 52 · 2025: 79 · 2024: 107 · 2023: 123 · 2022: 112 · 2021: 73 · 2020: 7
FWCI
74.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • IUScholarWorks (Indiana University) (submittedVersion, cc-by-nc)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by-nc)
  • The Mouseion at the JAXlibrary (Jackson Laboratory) (submittedVersion, cc-by-nc)
Funders
  • National Institutes of Health
  • National Center for Advancing Translational Sciences Institute (U24TR002306)
Topics
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Oncology › COVID-19 and healthcare impacts
Keywords
N3C, COVID-19 pandemic, open science, clinical data sharing, data harmonization, data governance, data quality assurance, synthetic data
MeSH terms
Intersectoral Collaboration, Data Science, Data Analysis, COVID-19, Humans, National Institutes of Health (U.S.), United States, Computer Security, Ethics Committees, Research, Government Regulation, Information Dissemination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review