Data Quality Assessment for Comparative Effectiveness Research in Distributed Data Networks
Open accessAbstract
BACKGROUND: Electronic health information routinely collected during health care delivery and reimbursement can help address the need for evidence about the real-world effectiveness, safety, and quality of medical care. Often, distributed networks that combine information from multiple sources are needed to generate this real-world evidence. OBJECTIVE: We provide a set of field-tested best practices and a set of recommendations for data quality checking for comparative effectiveness research (CER) in distributed data networks. METHODS: Explore the requirements for data quality checking and describe data quality approaches undertaken by several existing multi-site networks. RESULTS: There are no established standards regarding how to evaluate the quality of electronic health data for CER within distributed networks. Data checks of increasing complexity are often used, ranging from consistency with syntactic rules to evaluation of semantics and consistency within and across sites. Temporal trends within and across sites are widely used, as are checks of each data refresh or update. Rates of specific events and exposures by age group, sex, and month are also common. DISCUSSION: Secondary use of electronic health data for CER holds promise but is complex, especially in distributed data networks that incorporate periodic data refreshes. The viability of a learning health system is dependent on a robust understanding of the quality, validity, and optimal secondary uses of routinely collected electronic health data within distributed health data networks. Robust data quality checking can strengthen confidence in findings based on distributed data network.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Consistent methods for describing, assessing, and reporting DQ findings could help secondary data users and consumers understand the potential impact of DQ on reusing data and interpreting findings.
“consistent methods for describing, assessing, and reporting data quality (DQ) findings could be one way to help secondary data users and consumers understand the potential impact of DQ on reusing data and interpreting findings.36,41”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [36]
A substantial body of research suggests that data collected in EHRs and other operational systems may not be of sufficient quality for research.
“A substantial body of research suggests that data collected in EHRs and other operational systems may not be of sufficient quality for research.28–37”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Brown, P. (2002) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(02)00068-0 · Cited by 36 · see citing works
Flagged first, then by citations
Research works printed but not linked: 38
- [1]Federal Coordinating Council for Comparative Effectiveness Research Report to the President and Congress 2009 Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf 19567829
- [2]McClellan M Drug safety reform at the FDA--pendulum swing or systematic improvement? N Engl J Med 2007 356 1700 1702 17435081
- [4]Behrman RE Benner JS Brown JS Developing the Sentinel System--a national resource for evidence development. N Engl J Med 2011 364 498 499 21226658
- [5]Platt R Wilson M Chan KA The New Sentinel Network - Improving the Evidence of Medical-Product Safety. N Engl J Med 2009 361 645 647 19635947
- [6]Strom B Kimmel SE Hennessy S Pharmacoepidemiology 2012 Wiley West Sussex, England
- [7]Buntin MB Jain SH Blumenthal D Health information technology: laying the infrastructure for national health reform. Health Aff (Millwood) 2010 29 1214 1219 20530358
- [8]Olsen L IOM Roundtable on Evidence-Based Medicine. The Learning Healthcare System: Workshop Summary 2007 Washington, DC Available at 21452449
- [9]Institute of Medicine Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary 2011 The National Academies Press 22379651
- [10]Young PL Olsen L McGinnis JM Value in Health Care: Accounting for Cost, Quality, Safety, Outcomes, and Innovation: Workshop Summary 2010 The National Academies Press 21210558
- [12]Platt R Carnahan RM Brown JS The U.S. Food and Drug Administration's Mini-Sentinel program: status and direction. Pharmacoepidemiol Drug Saf 2012 21 Suppl 1 1 8 22262586
- [15]Maro JC Platt R Holmes JH Design of a National Distributed Health Data Network. Ann Intern Med 2009 151 5 341 4 [Epub ahead of print] 19638403
- [16]Moore KM Duddy A Braun MM Potential population-based electronic data sources for rapid pandemic influenza vaccine adverse event detection: a survey of health plans. Pharmacoepidemiol Drug Saf 2008 17 1137 1141 18763248
- [17]Toh S Platt R Steiner JF Comparative-effectiveness research in distributed health data networks. Clin Pharmacol Ther 2011 90 883 887 22030567
- [18]Yih WK Kulldorff M Fireman BH Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics 2011 127 Suppl 1 S54 64 21502252
- [20]Platt R Davis R Finkelstein J Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Center for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf 2001 10 373 377 11802579
- [21]Pace WD Cifuentes M Valuck RJ An electronic practice-based network for observational comparative effectiveness research. Ann Intern Med 2009 151 338 340 19638402
- [22]Wagner EH Greene SM Hart G Building a research consortium of large health systems: the Cancer Research Network. J Natl Cancer Inst Monogr 2005 3 11 16287880
- [23]Go AS Magid DJ Wells B The Cardiovascular Research Network: a new paradigm for cardiovascular quality and outcomes research. Circ Cardiovasc Qual Outcomes 2008 1 138 147 20031802
- [24]Robb MA Racoosin JA Sherman RE The US Food and Drug Administration's Sentinel Initiative: expanding the horizons of medical product safety. Pharmacoepidemiol Drug Saf 2012 21 Suppl 1 9 11 22262587
- [25]McGraw D Rosati K Evans B A policy framework for public health uses of electronic health data. Pharmacoepidemiol Drug Saf 2012 21 Suppl 1 18 22 22262589
- [26]Rosati K Using electronic health information for pharmacovigilance: the promise and the pitfalls. J Health Life Sci Law 2009 2 173 239 19673181
- [27]Salmon DA Akhtar A Mergler MJ Immunization-safety monitoring systems for the 2009 H1N1 monovalent influenza vaccination program. Pediatrics 2011 127 Suppl 1 S78 86 21502251
- [28]Velentgas P Bohn RL Brown JS A distributed research network model for post-marketing safety studies: the Meningococcal Vaccine Study. Pharmacoepidemiol Drug Saf 2008 17 1226 1234 18956428
- [29]Andrade S Raebel M Boudreau D Strom B Kimmel SE Hennessy S Chapter 12: Health Maintenance Organizations / Health Plans. Pharmacoepidemiology 2012 163 188 Wiley West Sussex, England
- [30]Hornbrook MC Hart G Ellis JL Building a virtual cancer research organization. J Natl Cancer Inst Monogr 2005 12 25 16287881
- [32]Chen RT Glasser JW Rhodes PH Vaccine Safety Datalink project: a new tool for improving vaccine safety monitoring in the United States. The Vaccine Safety Datalink Team. Pediatrics 1997 99 765 773 9164767
- [33]Davis RL Kolczak M Lewis E Active surveillance of vaccine safety: a system to detect early signs of adverse events. Epidemiology 2005 16 336 341 15824549
- [34]Schneeweiss S Avorn J A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol 2005 58 323 337 15862718
- [35]Suissa S Garbe E Primer: administrative health databases in observational studies of drug effects--advantages and disadvantages. Nat Clin Pract Rheumatol 2007 3 725 732 18037932
- [36]Kahn MG Raebel MA Glanz JM A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med Care 2012 50 Suppl S21 29 22692254 PMC3833692
- [37]Hall GC Sauer B Bourke A Guidelines for good database selection and use in pharmacoepidemiology research. Pharmacoepidemiol Drug Saf 2012 21 1 10 22069180
- [40]Observational Medical Outcomes Partnership. Generalized Review of OSCAR Unified Checking 2011 Available at: http://omop.fnih.org/GROUCH. June, 2012
- [41]Curtis LH Weiner MG Boudreau DM Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidemiol Drug Saf 2012 21 Suppl 1 23 31 22262590
- [44]Murphy SN Weber G Mendis M Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010 17 124 130 20190053 PMC3000779
- [45]Hennessy S Leonard CE Palumbo CM Quality of Medicaid and Medicare data obtained through Centers for Medicare and Medicaid Services (CMS). Med Care 2007 45 1216 1220 18007173
- [46]Hennessy S Bilker WB Weber A Descriptive analyses of the integrity of a US Medicaid claims database. Pharmacoepidemiol Drug Saf 2003 12 103 111 12647699
- [47]Brown PJ Warmington V Info-tsunami: surviving the storm with data quality probes. Inform Prim Care 2003 11 229 233 discussion 234-227 14980063
- [49]Brown PJ Harwood J Brantigan P Data quality probes--a synergistic method for quality monitoring of electronic medical record data accuracy and healthcare provision. Stud Health Technol Inform 2001 84 1116 1119 11604902
Cited by
Sorted by citations
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 636 · see citing works
Self-citationElectronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171 · see citing works
Qualls, L. G., Phillips, T. A., Hammill, B. G., Topping, J., Louzao, D. M., Brown, J. S., Curtis, L. H., & Marsolo, K. (2018) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.199 · Cited by 116 · see citing works
Self-citation
- Authors
- Brown, J. S., Kahn, M., & Toh, S.
- Affiliations
- Harvard University (US)
- Colorado Clinical and Translational Sciences Institute (US)
- Corresponding author
- Brown, Jeffrey S.
- Toh, Sengwee
- Venue
- Medical Care
- Published
- 2013-08
- Volume
- 51
- Issue
- Supplement 8Suppl 3
- Pages
- S22-S29
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC4306391
- OpenAlex
- W2065268028
- Cited by
- 131
- References
- 1
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 7 · 2023: 6 · 2022: 15 · 2021: 12 · 2020: 16 · 2019: 21 · 2018: 6 · 2017: 16 · 2016: 8 · 2015: 7 · 2014: 11 · 2013: 1
- FWCI
- 15.53
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000154)
- Agency for Healthcare Research and Quality (1r01hs019912-01, R01HS019908, R01HS019912)
- National Center for Advancing Translational Sciences (TR000154)
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Keywords
- data quality assessment, comparative effectiveness research, distributed data networks, electronic health data, secondary use of health data, data consistency checks, learning health systems, real-world evidence
- MeSH terms
- Data Collection, Humans, Research Design, Multicenter Studies as Topic, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
