Electronic health record data quality assessment and tools: a systematic review
Open accessThe DOI matches its Crossref record; 128 of the 128 references it cites were found in a registry or an index.
Abstract
Objective: We extended a 2013 literature review on electronic health record (EHR) data quality assessment approaches and tools to determine recent improvements or changes in EHR data quality assessment methodologies. Materials and Methods: We completed a systematic review of PubMed articles from 2013 to April 2023 that discussed the quality assessment of EHR data. We screened and reviewed papers for the dimensions and methods defined in the original 2013 manuscript. We categorized papers as data quality outcomes of interest, tools, or opinion pieces. We abstracted and defined additional themes and methods though an iterative review process. Results: We included 103 papers in the review, of which 73 were data quality outcomes of interest papers, 22 were tools, and 8 were opinion pieces. The most common dimension of data quality assessed was completeness, followed by correctness, concordance, plausibility, and currency. We abstracted conformance and bias as 2 additional dimensions of data quality and structural agreement as an additional methodology. Discussion: There has been an increase in EHR data quality assessment publications since the original 2013 review. Consistent dimensions of EHR data quality continue to be assessed across applications. Despite consistent patterns of assessment, there still does not exist a standard approach for assessing EHR data quality. Conclusion: Guidelines are needed for EHR data quality assessment to improve the efficiency, transparency, comparability, and interoperability of data quality assessment. These guidelines must be both scalable and flexible. Automation could be helpful in generalizing this process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Fifth, there are well-known issues when reusing EHR data for secondary analyses49. To address this, we applied a quality assessment framework to all variables50.”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2_v1 · Cited by 8,090
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“122 Patients may be assigned incorrect risk scores or be given incorrect treatment recommendations.”
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542 · cited 1×
“13 Although not considered a universal tool, this tool can be openly accessed and used to assess the quality of data conforming to the OMOP structure.”
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“120,121 These biased results may then have a negative impact on patient care when machine learning models are used in decision support tools in clinical practice.”
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 13×
“8 Similarly, there was an increase in the proportion of papers in which each method was used.”
InfluentialSelf-citationArtificial intelligence, bias and clinical safety
Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008370 · Cited by 956
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 697 · cited 3×
“7 This represents a shift in DQA practices since the 2013 review which only identified dimensions that focused on intrinsic and contextual DQA, a shift that may be partly due to the adoption of frameworks like the one presented by Kahn et al 9 and the increased adoption of CDMs both within research networks and collab…”
Self-citationAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415 · cited 1×
“Bias in EHR data can cause bias in the machine learning and artificial intelligence models developed using the data.(120,121) These biased results may then have a negative impact on patient care when machine learning models are used in decision support tools in clinical practice.”
Questions for artificial intelligence in health care
Maddox, T. M., Rumsfeld, J. S., & Payne, P. R. O. (2019) · JAMA · doi:10.1001/jama.2018.18932 · Cited by 385 · cited 1×
“The multitude of data available in EHRs make them wellsuited for high-dimensional analyses, including phenotyping as well as machine learning and artificial intelligence approaches.(2) Additionally, EHR data offer potential cost and time savings as an alternative to the primary collection of medical data for research…”
Self-citationDefining and measuring completeness of electronic health records for secondary use
Weiskopf, N. G., Hripcsak, G., Swaminathan, S., & Weng, C. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.010 · Cited by 367
Self-citationRichesson, R. L., Hammond, W. E., Nahm, M., Wixted, D., Simon, G. E., Robinson, J. G., Bauck, A. E., Cifelli, D., Smerek, M. M., Dickerson, J., et al. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001926 · Cited by 208
Challenges with quality of race and ethnicity data in observational databases
Polubriaginof, F. C. G., Ryan, P., Salmasian, H., Shapiro, A. W., Perotte, A., Safford, M. M., Hripcsak, G., Smith, S., Tatonetti, N. P., & Vawdrey, D. K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz113 · Cited by 208
A Data Quality Assessment Guideline for Electronic Health Record Data Reuse
Weiskopf, N. G., Bakken, S., Hripcsak, G., & Weng, C. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.218 · Cited by 152
Self-citationColquhoun, D. A., Shanks, A. M., Kapeles, S. R., Shah, N., Saager, L., Vaughn, M. T., Buehler, K., Burns, M. L., Tremper, K. K., Freundlich, R. E., et al. (2020) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004489 · Cited by 151
Wright, A., McCoy, A. B., Hickman, T.-T. T., Hilaire, D. S., Borbolla, D., Bowes, W. A., Dixon, W. G., Dorr, D. A., Krall, M., Malholtra, S., et al. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.06.011 · Cited by 150
Identifying periods of acceptable computer usage in primary care research databases
Horsfall, L., Walters, K., & Petersen, I. (2013) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3368 · Cited by 148
Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods
Feder, S. L. (2018) · Western Journal of Nursing Research · doi:10.1177/0193945916689084 · Cited by 147
Health informatics and EHR to support clinical research in the COVID-19 pandemic: an overview
Dagliati, A., Malovini, A. L., Tibollo, V., & Bellazzi, R. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa418 · Cited by 145
Daymont, C. B., Ross, M., Localio, A. R., Fiks, A. G., Wasserman, R. C., & Grundmeier, R. W. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx037 · Cited by 144
Sick patients have more data: the non-random completeness of electronic health records.
Weiskopf, N. G., Rusanov, A., & Weng, C. (2013) · Europe PMC (PubMed Central) · Cited by 132
Self-citationData Quality Assessment for Comparative Effectiveness Research in Distributed Data Networks
Brown, J. S., Kahn, M., & Toh, S. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1e2c · Cited by 131 · cited 3×
“11,53 Structured data lend themselves well to intrinsic and contextual approaches as they have likely been assessed for conform-ance, while unstructured data in the EHR or an associated source, such as clinical registries, may still require assessment of conformance.”
Brundin-Mather, R., Soo, A., Zuege, D. J., Niven, D. J., Fiest, K., Doig, C. J., Zygun, D., Boyd, J. M., Parsons Leigh, J., Bagshaw, S. M., et al. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.07.021 · Cited by 127
Coleman, N., Halas, G., Peeler, W., Casaclang, N., Williamson, T. S., & Katz, A. (2015) · BMC Family Practice · doi:10.1186/s12875-015-0223-z · Cited by 126
Challenges with Collecting Smoking Status in Electronic Health Records.
Polubriaginof, F. C. G., Salmasian, H., Albert, D. A., & Vawdrey, D. K. (2017) · PubMed · Cited by 121
Qualls, L. G., Phillips, T. A., Hammill, B. G., Topping, J., Louzao, D. M., Brown, J. S., Curtis, L. H., & Marsolo, K. (2018) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.199 · Cited by 116
Byrd, J. B., Vigen, R., Plomondon, M. E., Rumsfeld, J. S., Box, T. L., Fihn, S. D., & Maddox, T. M. (2013) · American Heart Journal · doi:10.1016/j.ahj.2012.12.009 · Cited by 115
Automated misspelling detection and correction in clinical free-text records
Lai, K. H., Topaz, M., Goss, F. R., & Zhou, L. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.04.008 · Cited by 113
Kohane, I. S., Aronow, B. J., Avillach, P., Beaulieu-Jones, B. K., Bellazzi, R., Bradford, R. L., Brat, G. A., Cannataro, M., Cimino, J. J., García-Barrio, N., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/22219 · Cited by 113
Use of EHRs data for clinical research: Historical progress and current applications
Nordo, A. H., Levaux, H. P., Becnel, L. B., Gálvez, J. L. V., Rao, P., Stem, K., Prakash, E., & Kush, R. D. (2019) · Learning Health Systems · doi:10.1002/lrh2.10076 · Cited by 110
Verma, A. A., Pasricha, S. V., Jung, H. Y., Kushnir, V., Mak, D. Y. F., Koppula, R., Guo, Y., Kwan, J. L., Lapointe-Shaw, L., Rawal, S., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa225 · Cited by 99
Pezoulas, V. C., Kourou, K. D., Kalatzis, F., Exarchos, T. P., Venetsanopoulou, A., Zampeli, E., Gandolfo, S., Skopouli, F., De Vita, S., Tzioufas, A. G., et al. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.03.001 · Cited by 96
Improving the quality of EHR recording in primary care: a data quality feedback tool
van der Bij, S., Khan, N., ten Veen, P., de Bakker, D. H., & Verheij, R. A. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw054 · Cited by 93
Biases introduced by filtering electronic health records for patients with “complete data.”
Weber, G. M., Adams, W. G., Bernstam, E. V., Bickel, J. P., Fox, K. P., Marsolo, K., Raghavan, V. A., Turchin, A., Zhou, X., Murphy, S. N., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx071 · Cited by 93
Sholle, E. T., Pinheiro, L. C., Adekkanattu, P., Davila, M. A., Johnson, S. B., Pathak, J., Sinha, S., Li, C., Lubansky, S. A., Safford, M. M., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz040 · Cited by 89
Hernandez‐Boussard, T., Monda, K. L., Crespo, B. C., & Riskin, D. J. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz119 · Cited by 89
Haskew, J., Rø, G., Saito, K., Turner, K., Odhiambo, G., Wamae, A., Sharif, S. K., & Sugishita, T. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.01.005 · Cited by 87
Data quality assessment framework to assess electronic medical record data for use in research
Reimer, A. P., Milinovich, A., & Madigan, E. A. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.03.006 · Cited by 86
Rhee, C., Jentzsch, M. S., Kadri, S. S., Seymour, C. W., Angus, D. C., Murphy, D. J., Martin, G. S., Dantes, R. B., Epstein, L., Fiore, A. E., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003554 · Cited by 85
A longitudinal analysis of data quality in a large pediatric data research network
Khare, R., Utidjian, L., Ruth, B. J., Kahn, M. G., Burrows, E., Marsolo, K., Patibandla, N., Razzaghi, H., Colvin, R., Ranade, D., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx033 · Cited by 81
A Framework for Data Quality Assessment in Clinical Research Datasets.
Lee, K., Weiskopf, N. G., & Pathak, J. (2017) · PubMed · Cited by 76
Self-citationMeasuring the rate of manual transcription error in outpatient point-of-care testing
Mays, J. A., & Mathias, P. C. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy170 · Cited by 76
Béresniak, A., Schmidt, A., Proeve, J., Bolaños, E., Patel, N., Ammour, N., Sundgren, M., Ericson, M., Karakoyun, T., Coorevits, P. L., et al. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.11.011 · Cited by 74
Pfaff, E. R., Girvin, A. T., Gabriel, D. L., Kostka, K., Morris, M., Palchuk, M. B., Lehmann, H. P., Amor, B., Bissell, M., Bradwell, K. R., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab217 · Cited by 73
Ammann, E. M., Kalsekar, I., Yoo, A., & Johnston, S. S. (2018) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4617 · Cited by 72
Multi-Institutional Sharing of Electronic Health Record Data to Assess Childhood Obesity
Bailey, L. C., Milov, D. E., Kelleher, K. J., Kahn, M. G., Beccaro, M. D., Yu, F., Richards, T. M., & Forrest, C. B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066192 · Cited by 67
A Data Quality Ontology for the Secondary Use of EHR Data.
(2015) · PubMed Central · Cited by 66
Lynch, K. E., Deppen, S. A., DuVall, S. L., Viernes, B., Cao, A., Park, D., Hanchrow, E. E., Hewa, K., Greaves, P., & Matheny, M. E. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-1697598 · Cited by 66
The impact of routine data quality assessments on electronic medical record data quality in Kenya
Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Prachi, M., Wamicwe, J., Odhiambo, J., Onyango, F., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195362 · Cited by 64
Application of an Ontology for Characterizing Data Quality for a Secondary Use of EHR Data
Speedie, S., Simon, G., Kumar, V., Westra, B., & Johnson, S. (2016) · Applied Clinical Informatics · doi:10.4338/aci-2015-08-ra-0107 · Cited by 63
Structured data quality reports to improve EHR data quality
Taggart, J., Liaw, S., & Yu, H. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.09.008 · Cited by 62
Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Datasets
Huser, V., DeFalco, F. J., Schuemie, M., Ryan, P. B., Shang, N., Velez, M., Park, R. W., Boyce, R. D., Duke, J., Khare, R., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1239 · Cited by 61
A basic model for assessing primary health care electronic medical record data quality
Terry, A. L., Stewart, M., Cejic, S., Marshall, J. N., de Lusignan, S., Chesworth, B. M., Chevendra, V., Maddocks, H., Shadd, J., Burge, F., et al. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0740-0 · Cited by 61
Discrepancies in Race and Ethnicity in the Electronic Health Record Compared to Self-report
Samalik, J. M., Goldberg, C. S., Modi, Z. J., Fredericks, E. M., Gadepalli, S. K., Eder, S. J., & Adler, J. (2022) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-022-01445-w · Cited by 61
Liaw, S., Taggart, J., Yu, H., de Lusignan, S., Kuziemsky, C. E., & Hayen, A. D. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.07.016 · Cited by 60
Dixon, B. E., Siegel, J. A., Oemig, T. V., & Grannis, S. J. (2013) · Public Health Reports® · doi:10.1177/003335491312800614 · Cited by 59
A clustering approach for detecting implausible observation values in electronic health records data
Estiri, H., Klann, J. G., & Murphy, S. N. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0852-6 · Cited by 59
Rahimi, A., Liaw, S., Taggart, J., RAY, P. K., & Yu, H. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.06.002 · Cited by 57
Assessing race and ethnicity data quality across cancer registries and EMRs in two hospitals
Lee, S. J. C., Grobe, J. E., & Tiro, J. A. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv156 · Cited by 57
An assessment of data quality in a multi-site electronic medical record system in Haiti
Puttkammer, N. H., Baseman, J. G., Devine, E. B., Valles, J. S., Hyppolite, N., Garilus, F. E., Honoré, J. G., Matheson, A. I., Zeliadt, S. B., Yuhas, K., et al. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.11.003 · Cited by 57
McVeigh, K. H., Newton-Dame, R., Chan, P. Y., Thorpe, L. E., Schreibstein, L., Tatem, K. S., Chernov, C., Lurie-Moroni, E., & Perlman, S. E. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1267 · Cited by 56
Singer, A. G., Yakubovich, S., Kroeker, A., Dufault, B., Duarte, R., & Katz, A. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw013 · Cited by 54
Electronic problem list documentation of chronic kidney disease and quality of care
Samal, L., Linder, J. A., Bates, D. W., & Wright, A. T. (2014) · BMC Nephrology · doi:10.1186/1471-2369-15-70 · Cited by 53
Yadav, S., Kazanji, N., K C, N., Paudel, S., Falatko, J., Shoichet, S., Maddens, M., & Barnes, M. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw067 · Cited by 52
Edmondson, M. E., & Reimer, A. P. (2020) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000609 · Cited by 52
Groenhof, T. K. J., Koers, L. R., Blasse, E., de Groot, M., Grobbee, D. E., Bots, M. L., Asselbergs, F. W., Lely, A. T., Haitjema, S., van Solinge, W. W., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.11.006 · Cited by 51
Just, B. H., Marc, D. T., Munns, M., & Sandefer, R. H. (2016) · PubMed Central · Cited by 50
Knake, L. A., Ahuja, M., McDonald, E. L., Ryckman, K. K., Weathers, N., Burstain, T., Dagle, J. M., Murray, J. C., & Nadkarni, P. (2016) · BMC Pediatrics · doi:10.1186/s12887-016-0592-z · Cited by 45
A Rule-Based Data Quality Assessment System for Electronic Health Record Data
Wang, Z., Talburt, J. R., Wu, N., Dagtas, S., & Zozus, M. N. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1715567 · Cited by 45
A Data Quality Assessment Guideline for Electronic Health Record Data Reuse
(2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/egems.1280 · Cited by 45
Dumas, R. P., Chreiman, K. M., Seamon, M. J., et al. (2018) · Injury · doi:10.1016/j.injury.2018.05.010 · Cited by 44
Parr, S. K., Shotwell, M. S., Jeffery, A. D., Lasko, T. A., & Matheny, M. E. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy110 · Cited by 44
Mind the clinical-analytic gap: Electronic health records and COVID-19 pandemic response
Sudat, S. E. K., Robinson, S. C., Mudiganti, S., Mani, A., & Pressman, A. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103715 · Cited by 44
Callahan, T. J., Barnard, J. G., Helmkamp, L. J., Maertens, J., & Kahn, M. G. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.214 · Cited by 41
Initializing a hospital-wide data quality program. The AP-HP experience.
Daniel, C., Serre, P., Orlova, N., Bréant, S., Paris, N., & Griffon, N. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.10.016 · Cited by 40
Moving Towards an EHR Data Quality Framework: The MIRACUM Approach
Kapsner, L. A., Kampf, M. O., Seuchter, S. A., Wabo, G. K., Gradinger, T., Ganslandt, T., Mate, S., Gruendner, J., Kraska, D., & Prokosch, H. (2019) · Studies in health technology and informatics · doi:10.3233/shti190834 · Cited by 40
Newton-Dame, R., McVeigh, K. H., Schreibstein, L., Perlman, S., Lurie-Moroni, L., Jacobson, L., Greene, C., Snell, E., & Thorpe, L. E. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1265 · Cited by 39
Liede, A., Hernandez, R. K., Roth, M., Calkins, G., Nicacio, L., & Larrabee, K. A. (2015) · Clinical Epidemiology · doi:10.2147/clep.s92209 · Cited by 37
Dziadkowiec, O., Callahan, T., Ozkaynak, M., Reeder, B., & Welton, J. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1201 · Cited by 37
Sirgo, G., Esteban, F., Gómez, J., Moreno, G., Rodríguez, A. H., Blanch, L. L., Guardiola, J., Gracia, R., de Haro, L., & Bodí, M. A. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.02.007 · Cited by 37
Ammann, E. M., Kalsekar, I., Yoo, A., Scamuffa, R., Hsiao, C.-W., Stokes, A. C., Morton, J. M., & Johnston, S. S. (2019) · Medicine · doi:10.1097/md.0000000000016438 · Cited by 37
Dentler, K., Ronald Cornet, A., ten Teije, A., Tanis, P. J., Klinkenbijl, J. H. G., Tytgat, K. M. A. J., & de Keizer, N. F. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-32 · Cited by 36
Kreuzthaler, M., Schulz, S., & Berghold, A. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.10.010 · Cited by 35
Scholte, M., van Dulmen, S. A., der Steen, C. N.-V., van der Wees, P. J., der Sanden, M. W. G. N., & Braspenning, J. C. C. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0382-4 · Cited by 35
Lee, W., Veeranki, S. P., Serag, H., Eschbach, K., & Smith, K. D. (2016) · PubMed Central · Cited by 33
Garcia, B. H., Djønne, B. S., Skjold, F., Mellingen, E. M., & Aag, T. I. (2017) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-017-0556-x · Cited by 33
Towards augmenting structured EHR data: a comparison of manual chart review and patient self-report.
Weiskopf, N. G., Cohen, A. M., Hannan, J., Jarmon, T., & Dorr, D. A. (2019) · PubMed · Cited by 33
Self-citationSidky, H., Young, J. C., Girvin, A. T., Lee, E., Shao, Y. R., Hotaling, N. A., Michael, S., Wilkins, K. J., Setoguchi, S., Funk, M. J., et al. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01839-2 · Cited by 33
Electronic Health Record Data Versus the National Health and Nutrition Examination Survey (NHANES)
Funk, L. M., Shan, Y., Voils, C. I., Kloke, J. D., & Hanrahan, L. P. (2017) · Medical Care · doi:10.1097/mlr.0000000000000693 · Cited by 32
Barkhuysen, P. N., de Grauw, W. J. C., Akkermans, R., Donkers, J., Schers, H., Biermans, M. C. J., Tiersma, W., & Peters, H. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001479 · Cited by 31
Real-World Evidence of COVID-19 Patients’ Data Quality in the Electronic Health Records
Binkheder, S., Asiri, M. A., Altowayan, K. W., Alshehri, T. M., Alzarie, M. F., Aldekhyyel, R. N., AlMaghlouth, I. A., & Almulhem, J. A. (2021) · Healthcare · doi:10.3390/healthcare9121648 · Cited by 31
Variation in Outcomes of Quality Measurement by Data Source
Angier, H. E., Gold, R. B., Gallia, C. A., Casciato, A., Tillotson, C. J., Marino, M., Mangione-Smith, R. M., & DeVoe, J. E. (2014) · PEDIATRICS · doi:10.1542/peds.2013-4277 · Cited by 30
Electronic health record problem lists: accurate enough for risk adjustment?
Daskivich, T. J., Abedi, G., Kaplan, S. H., Skarecky, D. W., Ahlering, T. E., Spiegel, B. M. R., Litwin, M. S., & Greenfield, S. (2018) · PubMed · Cited by 30
Thuraisingam, S., Chondros, P., Dowsey, M. M., Spelman, T. D., Garies, S., Choong, P., Gunn, J. M., & Manski-Nankervis, J.-A. E. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01669-6 · Cited by 30
Xi, N., Wallace, R., Agarwal, G., et al. (2015) · PubMed · Cited by 28
Sollie, A., Sijmons, R. H., Helsper, C. W., & Numans, M. E. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.08.004 · Cited by 28
Thompson, C. A., Jin, A., Luft, H. S., Lichtensztajn, D. Y., Allen, L., Liang, S., Schumacher, B. T., & Gomez, S. L. (2020) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-19-0882 · Cited by 28
A method for interoperable knowledge-based data quality assessment
Tute, E., Scheffner, I., & Marschollek, M. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01458-1 · Cited by 28
Chen, E. S., Carter, E. W., Sarkar, I. N., Winden, T., & Melton, G. B. (2014) · PubMed Central · Cited by 27
Ford, E., Carroll, J., Smith, H., Davies, K., Koeling, R., Petersen, I., Rait, G., & Cassell, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010393 · Cited by 26
Martin, S., Wagner, J., Lupulescu‐Mann, N., Ramsey, K., Cohen, A. A., Graven, P. F., Weiskopf, N. G., & Dorr, D. A. (2017) · Applied Clinical Informatics · doi:10.4338/aci-2016-12-ra-0210 · Cited by 26
Self-citationCarsley, S., Birken, C. S., Parkin, P. C., Pullenayegum, E., & Tu, K. (2018) · BMJ Health & Care Informatics · doi:10.14236/jhi.v25i1.963 · Cited by 25
A vision for the systematic monitoring and improvement of the quality of electronic health data.
Dixon, B. E., Rosenman, M. B., Xia, Y., & Grannis, S. J. (2013) · PubMed · Cited by 24
Horth, R., Wagstaff, S., Jeppson, T., Patel, V., McClellan, J., Bissonette, N., Friedrichs, M. D., & Dunn, A. C. (2019) · BMC Public Health · doi:10.1186/s12889-019-7367-z · Cited by 23
Cusick, M. M., Sholle, E., Davila, M. A., Kabariti, J., Cole, C. L., & Campion, T. R. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1718756 · Cited by 22
Engel, N., Wang, H., Jiang, X., Lau, C. Y., Patterson, J., Acharya, N., Beaton, M., Sulieman, L. M., Pavinkurve, N. P., & Natarajan, K. (2022) · PubMed · Cited by 22
Nicholson, B. D., Aveyard, P. N., Hamilton, W. T., Bankhead, C. R., Koshiaris, C., Stevens, S., Hobbs, R. F., & Perera, R. (2019) · Clinical Epidemiology · doi:10.2147/clep.s189989 · Cited by 20
Ostropolets, A., Reich, C., Ryan, P. B., Shang, N., Hripcsak, G. M., & Weng, C. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103363 · Cited by 19
Diaz‐Garelli, F., Strowd, R. E., III, Lawson, V., Mayorga, M. E., Wells, B. J., Lycan, T. W., & Topaloĝlu, Ü. (2020) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00114 · Cited by 19
Generative transfer learning for measuring plausibility of EHR diagnosis records
Estiri, H., Vasey, S., & Murphy, S. N. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa215 · Cited by 17
Lost in Translation: Diagnosis Records Show More Inaccuracies After Biopsy in Oncology Care EHRs.
Diaz‐Garelli, F., Strowd, R. E., III, Wells, B. J., Ahmed, T., Merrill, R., & Topaloĝlu, Ü. (2019) · PubMed · Cited by 16
Fu, S., Wen, A., Schaeferle, G. M., Wilson, P., Demuth, G. O., Ruan, X., Liu, S., Storlie, C. B., & Liu, H. (2022) · Unpublished · doi:10.13140/rg.2.2.29516.54400 · Cited by 16
Rockenschaub, P., Nguyen, V. G., Aldridge, R. W., Acosta, D., Garcia-Gomez, J. M., & Sáez, C. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034396 · Cited by 14
DQAgui: a graphical user interface for the MIRACUM data quality assessment tool
Mang, J. M., Seuchter, S. A., Gülden, C., Schild, S., Kraska, D., Prokosch, H., & Kapsner, L. A. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01961-z · Cited by 13
Quantifying Electronic Health Record Data Quality in Telehealth and Office-Based Diabetes Care
Wiley, K., Mendonça, E. A., Blackburn, J. L., Menachemi, N., de Groot, M., & Vest, J. R. (2022) · Applied Clinical Informatics · doi:10.1055/s-0042-1758737 · Cited by 13
Kiogou, S. D., Chi, C., Zhang, R., Ma, S., & Adam, T. J. (2022) · PubMed · Cited by 12
Ruddle, R. A., Adnan, M., & Hall, M. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-064887 · Cited by 12
Diaz‐Garelli, F., Strowd, R. E., III, Ahmed, T., Wells, B. J., Merrill, R., Laurini, J. A., Pasche, B., & Topaloĝlu, Ü. (2019) · JAMIA Open · doi:10.1093/jamiaopen/ooz020 · Cited by 11
Novel informatics approaches to COVID-19 Research: From methods to applications
Xu, H. X., Buckeridge, D. L., Wang, F., & Tarczy-Hornoch, P. Z. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104028 · Cited by 9
Making EHRs Trustable: A Quality Analysis of EHR-Derived Datasets for COVID-19 Research
Pedrera‐Jiménez, M., Barrio, N. G., Rubio-Mayo, P., de la Calle, G. M., Lalueza, A., García‐Reyne, A., Zamorro, M. J., Pons, A., Sanchez-Martin, M. J., Rojo, J. C., et al. (2022) · Studies in health technology and informatics · doi:10.3233/shti220430 · Cited by 9
Vass, A., Reinecke, I., Boeker, M., Prokosch, H., & Gülden, C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220046 · Cited by 9
Goldstein, J. E., Guo, X., Boland, M. V., & Smith, K. E. (2022) · Ophthalmology Science · doi:10.1016/j.xops.2022.100215 · Cited by 8
(2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/egems.1297 · Cited by 5
Cited by
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169
Rossetti, S. C., Dykes, P. C., Knaplund, C., Cho, S., Withall, J., Lowenthal, G., Albers, D., Lee, R. Y., Jia, H., Bakken, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03609-7 · Cited by 54
“Fifth, there are well-known issues when reusing EHR data for secondary analyses 49 .”
Pan, D., Nilsson, E., & Rahman Jabin, M. S. (2024) · Health Informatics Journal · doi:10.1177/14604582241270742 · Cited by 32
“35 However, it was evident in this study that these two dimensions had been jeopardised by various identi fi ed system issues.”
A Framework for Current and New Data Quality Dimensions: An Overview
Miller, R., Whelan, H., Chrubasik, M., Whittaker, D., Duncan, P., & Gregório, J. (2024) · Data · doi:10.3390/data9120151 · Cited by 33
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
Metadata
- Authors
- Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A.
- Affiliations
- Washington University in St. Louis (US)
- Oregon Health & Science University (US)
- Corresponding author
- Gupta, Aditi
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-06-30
- Volume
- 30
- Issue
- 10
- Pages
- 1730-1740
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10531113
- OpenAlex
- W4382753509
- Cited by
- 171
- References
- 128
- Citations by year
- 2026: 70 · 2025: 61 · 2024: 37 · 2023: 3
- FWCI
- 6.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by-nc)
- Funders
- National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092950)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Keywords
- electronic health records, data quality assessment, data quality dimensions, completeness, correctness, concordance, plausibility, currency, conformance, bias, data quality tools, interoperability, data quality guidelines
- MeSH terms
- Data Accuracy, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.