A Pragmatic Framework for Single-Site and Multisite Data Quality Assessment in Electronic Health Record-Based Clinical Research
Open accessAbstract
INTRODUCTION: Answers to clinical and public health research questions increasingly require aggregated data from multiple sites. Data from electronic health records and other clinical sources are useful for such studies, but require stringent quality assessment. Data quality assessment is particularly important in multisite studies to distinguish true variations in care from data quality problems. METHODS: We propose a "fit-for-use" conceptual model for data quality assessment and a process model for planning and conducting single-site and multisite data quality assessments. These approaches are illustrated using examples from prior multisite studies. APPROACH: Critical components of multisite data quality assessment include: thoughtful prioritization of variables and data quality dimensions for assessment; development and use of standardized approaches to data quality assessment that can improve data utility over time; iterative cycles of assessment within and between sites; targeting assessment toward data domains known to be vulnerable to quality problems; and detailed documentation of the rationale and outcomes of data quality assessments to inform data users. The assessment process requires constant communication between site-level data providers, data coordinating centers, and principal investigators. DISCUSSION: A conceptually based and systematically executed approach to data quality assessment is essential to achieve the potential of the electronic revolution in health care. High-quality data allow "learning health care organizations" to analyze and act on their own information, to compare their outcomes to peers, and to address critical scientific questions from the population perspective.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The current DQ literature is inconsistent in the use of terms that describe the complex multidimensional aspects of DQ.
“The current DQ literature is inconsistent in the use of terms that describe the complex multidimensional aspects of DQ.42–51”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672 · see citing works
Bleiholder, J., & Naumann, F. (2009) · ACM Computing Surveys · doi:10.1145/1456650.1456651 · Cited by 1,874 · see citing works
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724 · see citing works
Anchoring Data Quality Dimensions in Ontological Foundations
Wand, Y., & Wang, R. Y. (1996) · Communications of the ACM · doi:10.1145/240455.240479 · Cited by 1,479 · see citing works
Methodologies for data quality assessment and improvement
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541883 · Cited by 1,351 · see citing works
Gryna, F. M. (1970) · Cited by 893 · see citing works
Unwarranted variations in healthcare delivery: implications for academic medical centres
Wennberg, J. E. (2002) · BMJ · doi:10.1136/bmj.325.7370.961 · Cited by 813 · see citing works
A framework for analysis of data quality research
Wang, R., Storey, V. C., & Firth, C. (1995) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.404034 · Cited by 686 · see citing works
Data Quality : Concepts, Methodologies and Techniques
Batini, C., & Scannapieco, M. (2006) · doi:10.1007/3-540-33173-5 · Cited by 664 · see citing works
Arts, D. G. T. (2002) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1087 · Cited by 597 · see citing works
Tayi, G. K., & Ballou, D. P. (1998) · Communications of the ACM · doi:10.1145/269012.269021 · Cited by 436 · see citing works
Accuracy of Data in Computer-based Patient Records
Hogan, W. R., & Wagner, M. M. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040342 · Cited by 389 · see citing works
Chan, K. S., Fowles, J. B., & Weiner, J. P. (2010) · Medical Care Research and Review · doi:10.1177/1077558709359007 · Cited by 365 · see citing works
Chen, R. T., Glasser, J. W., Rhodes, P., Davis, R. L., Barlow, W. E., Thompson, R. S., Mullooly, J. P., BLACK, S. B., Shinefield, H. R., Vadheim, C. M., et al. (1997) · PEDIATRICS · doi:10.1542/peds.99.6.765 · Cited by 346 · see citing works
Quality Planning and Analysis: From Product Development Through Usage
Juran, J. M., & Gryna, F. M. (1982) · Medical Entomology and Zoology · Cited by 318 · see citing works
Redman, T. C. (2001) · Cited by 312 · see citing works
CONFOUNDING AND MISCLASSIFICATION
Greenland, S., & Robins, J. M. (1985) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a114131 · Cited by 232 · see citing works
Building a Research Consortium of Large Health Systems: The Cancer Research Network
Wagner, E. H. (2005) · JNCI Monographs · doi:10.1093/jncimonographs/lgi032 · Cited by 179 · see citing works
Risk of immune thrombocytopenic purpura after measles-mumps-rubella immunization in children
France, E. K., Glanz, J., Xu, S., Hambidge, S., Yamasaki, K., Black, S. B., Marcy, M., Mullooly, J. P., Jackson, L. A., Nordin, J., et al. (2008) · PEDIATRICS · doi:10.1542/peds.2007-1578 · Cited by 165 · see citing works
Design of a National Distributed Health Data Network
Maro, J. C., Platt, R. G., Holmes, J. H., Strom, B. L., Hennessy, S., Lazarus, R. M., & Brown, J. S. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-5-200909010-00139 · Cited by 165 · see citing works
Building a Virtual Cancer Research Organization
Hornbrook, M. C. (2005) · JNCI Monographs · doi:10.1093/jncimonographs/lgi033 · Cited by 154 · see citing works
Practice Variations And Health Care Reform: Connecting The Dots
Wennberg, J. E. (2004) · Health Affairs · doi:10.1377/hlthaff.var.140 · Cited by 150 · see citing works
Safety of Trivalent Inactivated Influenza Vaccine in Children 6 to 23 Months Old
Hambidge, S. J., Glanz, J. M., France, E. K., McClure, D. L., Xu, S., Yamasaki, K., Jackson, L. A., Mullooly, J. P., Zangwill, K. M., Marcy, S. M., et al. (2006) · JAMA · doi:10.1001/jama.296.16.1990 · Cited by 139 · see citing works
Self-citationToward Reuse of Clinical Data for Research and Quality Improvement: The End of the Beginning?
Weiner, M. G., & Embi, P. J. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-5-200909010-00141 · Cited by 129 · see citing works
Platt, R., Davis, R. L., Finkelstein, J. A., et al. (2001) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.607 · Cited by 81 · see citing works
Juran's Quality Planning and Analysis for Enterprise Quality
Gryna, F. M., Chua, R. C. H., & DeFeo, J. A. (2005) · Medical Entomology and Zoology · Cited by 80 · see citing works
Aronsky, D., & Haug, P. J. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070055 · Cited by 74 · see citing works
A Self-scaling, Distributed Information Architecture for Public Health, Research, and Clinical Care
McMurry, A., Gilbert, C., Reis, B. Y., et al. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2371 · Cited by 60 · see citing works
Future Directions for Small Area Variations
Wennberg, J. E. (1993) · Medical Care · doi:10.1097/00005650-199305001-00012 · Cited by 56 · see citing works
Addressing Variations: Is There Hope For The Future?
Wennberg, D. E., & Wennberg, J. E. (2003) · Health Affairs · doi:10.1377/hlthaff.w3.614 · Cited by 51 · see citing works
Glanz, J. M. (2011) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpediatrics.2011.112 · Cited by 43 · see citing works
Self-citationPredictive Value of ICD-9-CM Codes Used in Vaccine Safety Research
Donahue, J. G., Destefano, F. R., Baggs, J. M., Eriksen, E., Mullooly, J. P., & for the VSD Data Quality Working Group (2008) · Methods of Information in Medicine · doi:10.3414/me0500 · Cited by 38 · see citing works
A Hierarchical Approach to Improving Data Quality.
Abate, M. L., Diegert, K. V., & Allen, H. W. (1998) · Data Quality Journal · Cited by 37 · see citing works
Improving Data Quality of Health Information Systems: A Holistic Design- Oriented Approach
Mettler, T., Rohner, P., & Baacke, L. (2008) · Alexandria (UniSG) (University of St.Gallen) · Cited by 36 · see citing works
Distributed data processing for public health surveillance
Lazarus, R. M., Yih, K., & Platt, R. (2006) · BMC Public Health · doi:10.1186/1471-2458-6-235 · Cited by 31 · see citing works
Developing data production maps: meeting patient discharge data submission requirements
Davidson, B. N., Lee, Y. W., & Wang, R. (2004) · International Journal of Healthcare Technology and Management · doi:10.1504/ijhtm.2004.004978 · Cited by 28 · see citing works
Quality Assessments of HMO Diagnosis Databases Used to Monitor Childhood Vaccine Safety
Drew, L., DeStefano, F., Maher, J. E., et al. (2004) · Methods of Information in Medicine · doi:10.1055/s-0038-1633855 · Cited by 23 · see citing works
Moore, K. M., Duddy, A., Braun, M. M., et al. (2008) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1642 · Cited by 16 · see citing works
Think Research: Using Electronic Medical Records to Bridge Patient Care and Research
Anderson, S. M. (2005) · Cited by 16 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [22]Observational Medical Outcomes Partnership Generalized Review of OSCAR Unified Checking 2011 Available at: http://omop.fnih.org/GROUCH Accessed July 31, 2011
- [45]Riain C Helfert M An evaluation of data quality related problems patterns in healthcare information systems Paper presented at: Lisbon, Portugal IADIS virtual multi conference on computer science and information systems 2005
Cited by
Sorted by citations
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 636 · see citing works
Self-citationClinical Data Reuse or Secondary Use: Current Status and Potential Future Progress
Meystre, S. M., Lovis, C., Bürkle, T., Tognola, G., Budrionis, A., & Lehmann, C. U. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-007 · Cited by 204 · see citing works
A Data Quality Assessment Guideline for Electronic Health Record Data Reuse
Weiskopf, N. G., Bakken, S., Hripcsak, G., & Weng, C. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.218 · Cited by 152 · see citing works
Qualls, L. G., Phillips, T. A., Hammill, B. G., Topping, J., Louzao, D. M., Brown, J. S., Curtis, L. H., & Marsolo, K. (2018) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.199 · Cited by 116 · see citing works
Polnaszek, B., Gilmore-Bykovskyi, A., Hovanes, M., Roiland, R., Ferguson, P., Brown, R., & Kind, A. J. H. (2016) · Medical Care · doi:10.1097/mlr.0000000000000108 · Cited by 63 · see citing works
The impact of routine data quality assessments on electronic medical record data quality in Kenya
Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Prachi, M., Wamicwe, J., Odhiambo, J., Onyango, F., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195362 · Cited by 64 · see citing works
Sáez, C., Ferri, P., & García-Gómez, J. M. (2024) · Journal of Medical Internet Research · doi:10.2196/50295 · Cited by 36 · see citing works
Data Quality in Clinical Research
Nahm, M. (2012) · Health Informatics · doi:10.1007/978-1-84882-448-5_10 · Cited by 32 · see citing works
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37 · see citing works
Fu, A., Shen, T., Roberts, S. B., Liu, W., Vaidyanathan, S., Marchena-Romero, K.-J., Lam, Y. Y. P., Shah, K., Mak, D. Y. F., GEMINI Investigators, et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf042 · Cited by 14 · see citing works
Souza, J., Caballero, I., Vasco Santos, J., Lobo, M., Pinto, A., Viana, J., Sáez, C., Lopes, F., & Freitas, A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104242 · Cited by 7 · see citing works
- Authors
- Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F.
- Affiliations
- Colorado Clinical and Translational Sciences Institute (US)
- Kaiser Permanente (US)
- Kaiser Permanente Center for Health Research (US)
- Corresponding author
- Kahn, Michael G.
- Venue
- Medical Care
- Published
- 2012-07
- Volume
- 50
- Pages
- S21-S29
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3833692
- OpenAlex
- W2134256069
- Cited by
- 260
- References
- 39
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 15 · 2023: 20 · 2022: 14 · 2021: 10 · 2020: 19 · 2019: 24 · 2018: 23 · 2017: 24 · 2016: 21 · 2015: 21 · 2014: 22 · 2013: 17 · 2012: 4
- FWCI
- 33.27
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Agency for Healthcare Research and Quality (1r01hs019912-01, 1R01HS019908, R01HS019908, R01HS019912)
- National Center for Research Resources (UL1RR025780)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Keywords
- data quality assessment, electronic health records, data quality dimensions, data coordinating center, data utility, data prioritization
- MeSH terms
- Cooperative Behavior, Humans, Medical Informatics, Outcome and Process Assessment, Health Care, Quality Assurance, Health Care, United States, Patient-Centered Care, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
