Validity of electronic health record-derived quality measurement for performance monitoring
Open accessAbstract
BACKGROUND: Since 2007, New York City's primary care information project has assisted over 3000 providers to adopt and use a prevention-oriented electronic health record (EHR). Participating practices were taught to re-adjust their workflows to use the EHR built-in population health monitoring tools, including automated quality measures, patient registries and a clinical decision support system. Practices received a comprehensive suite of technical assistance, which included quality improvement, EHR customization and configuration, privacy and security training, and revenue cycle optimization. These services were aimed at helping providers understand how to use their EHR to track and improve the quality of care delivered to patients. MATERIALS AND METHODS: Retrospective electronic chart reviews of 4081 patient records across 57 practices were analyzed to determine the validity of EHR-derived quality measures and documented preventive services. RESULTS: Results from this study show that workflow and documentation habits have a profound impact on EHR-derived quality measures. Compared with the manual review of electronic charts, EHR-derived measures can undercount practice performance, with a disproportionately negative impact on the number of patients captured as receiving a clinical preventive service or meeting a recommended treatment goal. CONCLUSION: This study provides a cautionary note in using EHR-derived measurement for public reporting of provider performance or use for payment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
EHRs typically are optimized for efficient patient care and nonclinical administrative requirements, resulting in great variation in clinical documentation practices, even among users of the same systems.
“EHRs typically are optimized for efficient patient care and nonclinical administrative requirements, resulting in great variation in clinical documentation practices, even among users of the same systems.38–40”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [40]
The Discussion (page not provided) reports wide variation in documentation between providers, and the study examined practices using the same EHR system. It does not establish that EHRs are typically optimized for efficient patient care and nonclinical administrative requirements, or that such optimization causes the variation. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Chan, K. S., Fowles, J. B., & Weiner, J. P. (2010) · Medical Care Research and Review · doi:10.1177/1077558709359007 · Cited by 365 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [1]Blumenthal D Promoting use of health IT: why be a meaningful user? Md Med 2010; 11: 18– 19 21067108
- [3]Frieden TR Mostashari F Health care as if health mattered. JAMA 2008; 299: 950– 2 18314438
- [4]Jollis JG Ancukiewicz M DeLong ER Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann Intern Med 1993; 119: 844– 50 8018127
- [5]Tang PC Ralston M Arrigotti MF Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc 2007; 14: 10– 15 17068349 PMC2215069
- [6]Roski J McClellan M Measuring health care performance now, not tomorrow: essential steps to support effective health reform. Health Aff (Millwood) 2011; 30: 682– 9 21471489
- [7]Kahn M Ranade D The impact of electronic medical records data sources on an adverse drug event quality measure. J Am Med Inform Assoc 2010; 17: 185– 91 20190062 PMC3000787
- [9]Galanter WL Hier DB Jao C Computerized physician order entry of medications and clinical decision support can improve problem list documentation compliance. Int J Med Inform 2010; 79: 332– 8 18599342
Cited by
Sorted by citations
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 636 · see citing works
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · see citing works
Electronic health records-driven phenotyping: challenges, recent advances, and perspectives
Pathak, J., Kho, A. N., & Denny, J. C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002428 · Cited by 274 · see citing works
Ferreira, J. J., Godinho, C., Santos, A. T., Domingos, J., Abreu, D., Lobo, R., Gonçalves, N., Barra, M., Larsen, F., Fagerbakke, Ø., et al. (2015) · BMC Neurology · doi:10.1186/s12883-015-0343-z · Cited by 96 · see citing works
The impact of routine data quality assessments on electronic medical record data quality in Kenya
Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Prachi, M., Wamicwe, J., Odhiambo, J., Onyango, F., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195362 · Cited by 64 · see citing works
Haberer, J. E., Trabin, T., & Klinkman, M. (2013) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2013.03.009 · Cited by 39 · see citing works
Chasm Between Cancer Quality Measures and Electronic Health Record Data Quality
Schorer, A. E., Moldwin, R., Koskimaki, J., Bernstam, E. V., Venepalli, N. K., Miller, R. S., & Chen, J. L. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00128 · Cited by 12 · see citing works
- Authors
- Parsons, A., McCullough, C., Wang, J., & Shih, S.
- Affiliations
- New York City Department of Health and Mental Hygiene (US)
- Corresponding author
- Parsons, A.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-07-01
- Volume
- 19
- Issue
- 4
- Pages
- 604-609
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC3384112
- OpenAlex
- W2122019985
- Cited by
- 137
- References
- 1
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 4 · 2023: 6 · 2022: 4 · 2021: 7 · 2020: 1 · 2019: 7 · 2018: 10 · 2017: 8 · 2016: 15 · 2015: 18 · 2014: 22 · 2013: 24 · 2012: 1
- FWCI
- 33.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- Agency for Healthcare Research and Quality (R18HS17059, R18HS17294, R18HS017059)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › General Health Professions › Primary Care and Health Outcomes
- Keywords
- electronic health records, quality metrics, measurement validity, primary care, preventive services, clinical decision support, population health monitoring, workflow
- MeSH terms
- Adult, Documentation, Female, Humans, Male, Medical Audit, Middle Aged, New York City, Primary Health Care, Quality Assurance, Health Care, Reimbursement, Incentive, Retrospective Studies, Reproducibility of Results, Information Storage and Retrieval, Benchmarking, Workflow, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
