Review: Electronic Health Records and the Reliability and Validity of Quality Measures: A Review of the Literature
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
Previous reviews of research on electronic health record (EHR) data quality have not focused on the needs of quality measurement. The authors reviewed empirical studies of EHR data quality, published from January 2004, with an emphasis on data attributes relevant to quality measurement. Many of the 35 studies reviewed examined multiple aspects of data quality. Sixty-six percent evaluated data accuracy, 57% data completeness, and 23% data comparability. The diversity in data element, study setting, population, health condition, and EHR system studied within this body of literature made drawing specific conclusions regarding EHR data quality challenging. Future research should focus on the quality of data from specific EHR components and important data attributes for quality measurement such as granularity, timeliness, and comparability. Finally, factors associated with poor or variability in data quality need to be better understood and effective interventions developed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In a 2010 review, Chan et al 10 looked at the quality of the same clinical concepts across multiple institutions, and still found a great deal of variability. The completeness of blood pressure recordings, for example, fell anywhere between 0.1% and 51%.”
From Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research (2013) · cites this paper as [10]
In the citing paper
“EHRs typically are optimized for efficient patient care and nonclinical administrative requirements, resulting in great variation in clinical documentation practices, even among users of the same systems.38–40”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [39]
In the citing paper
“In particular, representativeness is often not a factor86,87 while it is a crucial property for secondary use of data in clinical studies88 or when reusing medical data as training data for ML applications.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [86]
In the citing paper
“While these developments have advanced the understanding of data quality in the context of medical applications, frameworks for EHR frequently focus on the data quality of individual patients86,87, neglecting data quality aspects for the overall population.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [86]
Citations
Cites
The American Journal of Managed Care
(2020) · The American Journal of Managed Care · doi:10.37765/ajmc · Cited by 3,237
Electronic health records in ambulatory care—a national survey of physicians
DesRoches, C. M., Campbell, E. G., Rao, S. R., Donelan, K., Ferris, T. G., Jha, A., Kaushal, R., Levy, D. E., Rosenbaum, S., Shields, A. E., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa0802005 · Cited by 1,111 · cited 6×
“Wherever possible, we organize study findings for each data attribute by specific data element, major EHR functions (e.g., problem or medication list, computerized physician order entry; desRoches et al., 2008), and the data flow process (data documentation, data abstraction, data aggregation across systems).”
BackgroundMethodInfluentialInformatics in Primary Care: Strategies in Information Management for the Healthcare Provider
(2002) · Health Informatics · doi:10.1007/978-1-4613-0069-4 · Cited by 422
Systematic review of scope and quality of electronic patient record data in primary care
Thiru, K., Hassey, A., & Sullivan, F. (2003) · BMJ · doi:10.1136/bmj.326.7398.1070 · Cited by 342
Agreement between patient-reported symptoms and their documentation in the medical record.
Pakhomov, S., Jacobsen, S. J., Chute, C. G., & Roger, V. L. (2008) · PubMed · Cited by 336
Rewarding Provider Performance: Aligning Incentives in Medicare
English, W. J. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-8-200804150-00028 · Cited by 220
Tang, P. C., Ralston, M., Arrigotti, M. F., Qureshi, L., & Graham, J. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2198 · Cited by 193 · cited 4×
“It has also been suggested that the clinical relevance of the data and their anticipated future use (e.g., for decision support) may affect data quality (Roukema et al., 2006; Tang et al., 2007).”
BackgroundResultInfluentialEgger, M. (2008) · Bulletin of the World Health Organization · doi:10.2471/blt.07.049908 · Cited by 177
Review: Use of Electronic Medical Records for Health Outcomes Research: A Literature Review
Dean, B. B., Lam, J., Natoli, J. L., Butler, Q., Aguilar, D., & Nordyke, R. J. (2009) · Medical Care Research and Review · doi:10.1177/1077558709332440 · Cited by 176
Pringle, M., Ward, P., & Chilvers, C. E. D. (1995) · Europe PMC (PubMed Central) · Cited by 146
Assessing the accuracy of computerized medication histories.
Kaboli, P. J., McClimon, B. J., Hoth, A. B., & Barnett, M. J. (2004) · PubMed · Cited by 120
Conroy, M., Majchrzak, N., Silverman, C., Chang, Y., Regan, S., Schneider, L., & Rigotti, N. (2005) · Nicotine & Tobacco Research · doi:10.1080/14622200500078089 · Cited by 119
Enhancing Performance Measurement: NCQA's Road Map for a Health Information Framework
Schneider, E. C., Riehl, V., Courte-Wienecke, S., Eddy, D. M., & Sennett, C. (1999) · JAMA · doi:10.1001/jama.282.12.1184 · Cited by 114
Improving Medication Reconciliation in the Outpatient Setting
Varkey, P., Cunningham, J. L., & Bisping, S. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33033-x · Cited by 113
Quality Improvement with an Electronic Health Record: Achievable, but Not Automatic
Baron, R. J. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00007 · Cited by 108
Copying and pasting of examinations within the electronic medical record
Thielke, S. M., Hammond, K. W., & Helbig, S. (2006) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.06.004 · Cited by 101
Paper Versus Computer: Feasibility of an Electronic Medical Record in General Pediatrics
Roukema, J., Los, R. K., Bleeker, S. E., van Ginneken, A. M., van der Lei, J., & Moll, H. A. (2006) · Pediatrics · doi:10.1542/peds.2004-2741 · Cited by 101
Baker, D. W., Persell, S. D., Thompson, J. A., Soman, N. S., Burgner, K. M., Liss, D., & Kmetik, K. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-4-200702200-00006 · Cited by 89
Staroselsky, M., Volk, L. A., Tsurikova, R., Newmark, L. P., Lippincott, M. F., Litvak, I., Kittler, A. F., Wang, T., Wald, J. S., & Bates, D. W. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.03.001 · Cited by 88
Hazlehurst, B. L., Sittig, D. F., Stevens, V. J., SMITH, K., Hollis, J. F., VOGT, T. M., Winickoff, J. P., GLASGOW, R., Palen, T. E., & Rigotti, N. A. (2005) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2005.08.007 · Cited by 87
Persell, S. D. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.20.2272 · Cited by 79
Staroselsky, M., Volk, L. A., Tsurikova, R., Pizziferri, L., Lippincott, M., Wald, J., & Bates, D. W. (2006) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.10.004 · Cited by 76
Goulet, J. L., Erdos, J., Kancir, S., Levin, F. L., Wright, S. M., Daniels, S. M., Nilan, L., & Justice, A. C. (2007) · Medical Care · doi:10.1097/01.mlr.0000244510.09001.e5 · Cited by 67
Sharing electronic health records: the patient view
Powell, J., Fitton, R., & Fitton, C. (2006) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v14i1.614 · Cited by 63
Benin, A. L., Vitkauskas, G., Thornquist, E., Shapiro, E. D., Concato, J., Aslan, M., & Krumholz, H. M. (2005) · Medical Care · doi:10.1097/01.mlr.0000167185.26058.8e · Cited by 52
Medication Reconciliation in Ambulatory Oncology
Weingart, S. N., Cleary, A. M., Seger, A. C., Eng, T. K., Saadeh, M., Gross, A., & Shulman, L. N. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33090-0 · Cited by 50
Electronic Health Record-Based Cardiac Risk Assessment and Identification of Unmet Preventive Needs
Persell, S. D., Dunne, A. P., Lloyd‐Jones, D. M., & Baker, D. W. (2009) · Medical Care · doi:10.1097/mlr.0b013e31818dce21 · Cited by 50
Linder, J. A., Kaleba, E. O., & Kmetik, K. S. (2009) · Medical Care · doi:10.1097/mlr.0b013e318189375f · Cited by 45
de Lusignan, S., Hague, N., Brown, A., & Majeed, A. (2004) · Journal of Public Health · doi:10.1093/pubmed/fdh104 · Cited by 43
Faulconer, E., & DeLusignan, S. (2004) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v12i4.132 · Cited by 41
Use of Electronic Medical Record Data for Quality Improvement in Schizophrenia Treatment
Owen, R. R., Thrush, C. R., Cannon, D., Sloan, K. L., Curran, G., Hudson, T., Austen, M., & Ritchie, M. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1498 · Cited by 34
Improving Hypertension Quality Measurement Using Electronic Health Records
Persell, S. D., Kho, A., Thompson, J. A., & Baker, D. W. (2009) · Medical Care · doi:10.1097/mlr.0b013e31818b070c · Cited by 30
BRIEF REPORT: Failure of an Electronic Medical Record Tool to Improve Pain Assessment Documentation
Saigh, O., Triola, M. M., & Link, R. N. (2006) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2006.00330.x · Cited by 27
Quality Performance Measurement Using the Text of Electronic Medical Records
Pakhomov, S., Bjornsen, S. S., Hanson, P., & Smith, S. A. (2008) · Medical Decision Making · doi:10.1177/0272989x08315253 · Cited by 27
Evaluating Healthcare Quality Using Natural Language Processing
Baldwin, K. B. (2008) · Journal for Healthcare Quality · doi:10.1111/j.1945-1474.2008.tb01151.x · Cited by 25
Staes, C. J., Bennett, S. T., Evans, R. S., Narus, S. P., Huff, S. M., & Sorensen, J. B. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1813 · Cited by 24 · cited 3×
“521 same organization and using the same EHR, may rely on electronically linked data sources such as commercial laboratories to varying degrees, particularly if their patients have different health plans or are geographically dispersed (Staes et al., 2006).”
BackgroundOpportunities and Challenges for Measuring Cost, Quality, and Clinical Effectiveness in Health Care
Fishman, P. A., Hornbrook, M. C., Meenan, R. T., & Goodman, M. J. (2004) · Medical Care Research and Review · doi:10.1177/1077558704267512 · Cited by 24
Measuring quality of care for rheumatic diseases using an electronic medical record
Agnew-Blais, J., Coblyn, J., Katz, J. N., Anderson, R. J., Mehta, J., & Solomon, D. (2008) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2008.089318 · Cited by 23
de Lusignan, S., Chan, T., Wood, O., Hague, N., Valentin, T., & Van Vlymen, J. (2005) · Public Health · doi:10.1016/j.puhe.2004.10.018 · Cited by 19
McGinnis, K. A., Skanderson, M., Levin, F. L., Brandt, C., Erdos, J., & Justice, A. C. (2008) · Medical Care · doi:10.1097/mlr.0b013e31817d69c2 · Cited by 19
Goodyear‐Smith, F., Grant, C. C., York, D. G., Kenealy, T. W., Copp, J., Petousis‐Harris, H., Turner, N. M., & Kerse, N. M. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.008 · Cited by 18
Measuring the Completeness and Currency of Codified Clinical Information
Williams, J. G. (2003) · Methods of Information in Medicine · doi:10.1055/s-0038-1634243 · Cited by 18
Prenatal Syphilis Screening Rates Measured Using Medicaid Claims and Electronic Medical Records
Rosenman, M. B., Tao, G., Szucs, K. A., Wang, J., Ambuehl, R. J., & Mahon, B. E. (2008) · Sexually Transmitted Diseases · doi:10.1097/olq.0b013e31815fa5bb · Cited by 15
Clinical terminology support for a national ambulatory practice outcomes research network.
Ricciardi, T. N., Lieberman, M. I., Kahn, M. G., & Masarie, F. E. (2005) · PubMed Central · Cited by 10
Integration challenges of clinical information systems developed without a shared data dictionary.
Hicken, V. N., Thornton, S. N., & Rocha, R. A. (2004) · PubMed · Cited by 9
Improving the Documentation of Vital Signs: A Business Reengineering Efficiency Study
Szpunar, S. M., Williams, P. A. H., Dagroso, D., Enberg, R. N., & Chesney, J. D. (2008) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(08)34020-3 · Cited by 4
Research works printed but not linked: 7
- Faulconer, E.R., 2004, Informatics in Primary Care, 12, 243
- Hicken, V.N., 2004, Studies in Health Technology and Informatics, 107, 1053
- Kaboli, P.J., 2004, American Journal of Managed Care, 10, 872
- Powell, J., 2006, Informatics in Primary Care, 14, 55
- Pringle, M., 1995, British Journal of General Practice, 45, 537
- Ricciardi, T.N., AMIA Annual Symposium Proceedings
- 10.1007/s11606-006-0256-z · doi:10.1007/s11606-006-0256-z
Cited by
The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216
“Hogan and Wagner, 8 in their 1997 literature review, found 22 that the correctness of data ranged between 44% and 100%, and completeness between 1.1% and 100%, depending the clinical concepts being studied.”
BackgroundInfluentialKahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 697
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567
The impact of electronic health records on healthcare quality: a systematic review and meta-analysis
Campanella, P., Lovato, E., Marone, C., Fallacara, L., Mancuso, A., Ricciardi, W., & Specchia, M. L. (2016) · The European Journal of Public Health · doi:10.1093/eurpub/ckv122 · Cited by 434
Metadata
- Authors
- Chan, K. S., Fowles, J. B., & Weiner, J. P.
- Affiliations
- Johns Hopkins University (US)
- Park Nicollet Institute, Minneapolis, MN
- Venue
- Medical Care Research and Review
- Publisher
- SAGE Publications
- Published
- 2010-02-11
- Volume
- 67
- Issue
- 5
- Pages
- 503-527
- ISSN
- 1077-5587 (ISSN-L)
- 1552-6801
- OpenAlex
- W2135703887
- Cited by
- 365
- References
- 47
- Citations by year
- 2026: 10 · 2025: 14 · 2024: 19 · 2023: 20 · 2022: 31 · 2021: 36 · 2020: 23 · 2019: 37 · 2018: 28 · 2017: 19 · 2016: 22 · 2015: 26 · 2014: 24 · 2013: 25 · 2012: 18
- FWCI
- 2.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Medical Care Research and Review (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Keywords
- electronic health records, EHR data quality, quality metrics, data accuracy, data completeness, data comparability, data granularity, timeliness, reliability statistics, validity
- MeSH terms
- Humans, Quality Assurance, Health Care, Research Design, United States, Reproducibility of Results, Quality Indicators, Health Care, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.