CoChat

Automating Electronic Health Record Data Quality Assessment

Open access
Ozonze, Obinwa, Scott, Philip J., Hopgood, Adrian A. · 2023 · Journal of Medical Systems
Cited by 69

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Abstract

Information systems such as Electronic Health Record (EHR) systems are susceptible to data quality (DQ) issues. Given the growing importance of EHR data, there is an increasing demand for strategies and tools to help ensure that available data are fit for use. However, developing reliable data quality assessment (DQA) tools necessary for guiding and evaluating improvement efforts has remained a fundamental challenge. This review examines the state of research on operationalising EHR DQA, mainly automated tooling, and highlights necessary considerations for future implementations. We reviewed 1841 articles from PubMed, Web of Science, and Scopus published between 2011 and 2021. 23 DQA programs deployed in real-world settings to assess EHR data quality (n = 14), and a few experimental prototypes (n = 9), were identified. Many of these programs investigate completeness (n = 15) and value conformance (n = 12) quality dimensions and are backed by knowledge items gathered from domain experts (n = 9), literature reviews and existing DQ measurements (n = 3). A few DQA programs also explore the feasibility of using data-driven techniques to assess EHR data quality automatically. Overall, the automation of EHR DQA is gaining traction, but current efforts are fragmented and not backed by relevant theory. Existing programs also vary in scope, type of data supported, and how measurements are sourced. There is a need to standardise programs for assessing EHR data quality, as current evidence suggests their quality may be unknown.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10916-022-01892-2

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 85

  1. Warren, L.R., et al., Improving data sharing between acute hospitals in England: an overview of health record system distribution and retrospective observational analysis of inter-hospital transitions of care. BMJ Open, 2019. 9(12): p. e031637. · DOI · Paper page
  2. Atasoy, H., B.N. Greenwood, and J.S. McCullough, The Digitization of Patient Care: A Review of the Effects of Electronic Health Records on Health Care Quality and Utilization. Annu Rev Public Health, 2019. 40: p. 487–500. · DOI · Paper page
  3. Hayrinen, K., K. Saranto, and P. Nykanen, Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform, 2008. 77(5): p. 291–304. · DOI · Paper page
  4. Meystre, S.M., et al., Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress. Yearb Med Inform, 2017. 26(1): p. 38–52. · DOI · Paper page
  5. Topol, E., The Topol Review Preparing the Healthcare Workforce to Deliver the Digital Future, 2019: p. 1–48.
  6. Safran, C., Update on Data Reuse in Health Care. Yearb Med Inform, 2017. 26(1): p. 24–27. · doi:10.15265/iy-2017-013
  7. Vuokko, R., et al., Secondary Use of Structured Patient Data: Interim Results of A Systematic Review. 2015. p. 291–295.
  8. Bayley, K.B., et al., Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med Care, 2013. 51(8 Suppl 3): p. S80-6.
  9. WHO, Administrative Errors: Technical Series on Safer Primary Care, in WHO Press. 2016: Geneva.
  10. Ghosh, A., S. McCarthy, and E. Halcomb, Perceptions of primary care staff on a regional data quality intervention in Australian general practice: A qualitative study. BMC Family Practice, 2016. 17(1). · DOI
  11. Salomon, R.M., et al., Openness of patients’ reporting with use of electronic records: Psychiatric clinicians’ views. Journal of the American Medical Informatics Association, 2010. 17(1): p. 54–60. · DOI · Paper page
  12. Peivandi, S., et al., Evaluation and comparison of errors on nursing notes created by online and offline speech recognition technology and handwritten: an interventional study. BMC Medical Informatics and Decision Making, 2022. 22(1): p. 96. · DOI
  13. Colin, N.V., et al., Understanding the Impact of Variations in Measurement Period Reporting for Electronic Clinical Quality Measures. EGEMS (Wash DC), 2018. 6(1): p. 17.
  14. Bowman, S., Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag, 2013. 10: p. 1c-1c.
  15. O’Donnell, H.C., et al., Physicians’ Attitudes Towards Copy and Pasting in Electronic Note Writing. Journal of General Internal Medicine, 2009. 24(1): p. 63–68. · DOI
  16. Coleman, N., et al., From patient care to research: A validation study examining the factors contributing to data quality in a primary care electronic medical record database. BMC Family Practice, 2015. 16(1). · DOI
  17. Economic analysis of the prevalence and clinical and economic burden of medication error in England BMJ Quality & Safety, 2021. 30(2): p. 96. · DOI · Paper page
  18. Zozus, M.N., M. Penning, and W.E. Hammond, Factors impacting physician use of information charted by others. JAMIA Open, 2019. 2(1): p. 107–114. · DOI
  19. Munyisia, E.N., D. Reid, and P. Yu, Accuracy of outpatient service data for activity-based funding in New South Wales, Australia. Health Inf Manag, 2017. 46(2): p. 78–86.
  20. Kaplan, B., How Should Health Data Be Used?: Privacy, Secondary Use, and Big Data Sales. Cambridge Quarterly of Healthcare Ethics, 2016. 25(2): p. 312–329. · DOI · Paper page
  21. Nouraei, S.A.R., et al., Accuracy of clinician-clinical coder information handover following acute medical admissions: Implication for using administrative datasets in clinical outcomes management. Journal of Public Health (United Kingdom), 2016. 38(2): p. 352–362.
  22. Feldman, K., et al., Beyond volume: The impact of complex healthcare data on the machine learning pipeline Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017. 10344 LNAI: p. 150–169. · DOI
  23. Hanauer, D.A., et al., Complexities, variations, and errors of numbering within clinical notes: The potential impact on information extraction and cohort-identification. BMC Medical Informatics and Decision Making, 2019. 19(Suppl 3): p. 75–75. · DOI
  24. Batini, C., et al., Methodologies for data quality assessment and improvement. ACM computing surveys (CSUR), 2009. 41(3): p. 16. · DOI · Paper page
  25. Veiga, A.K., et al., A conceptual framework for quality assessment and management of biodiversity data. PLoS ONE, 2017. 12(6): p. e0178731-e0178731. · DOI
  26. Kelly, M.M., R.J. Coller, and P.L. Hoonakker, Inpatient Portals for Hospitalized Patients and Caregivers: A Systematic Review. J Hosp Med, 2018. 13(6): p. 405–412. · DOI
  27. Wiebe, N., et al., Evaluation of interventions to improve inpatient hospital documentation within electronic health records: a systematic review. J Am Med Inform Assoc, 2019. 26(11): p. 1389–1400. · DOI · Paper page
  28. Isaksen, H., et al., Design of tooltips for data fields: A field experiment of logging use of tooltips and data correctness. 2017. p. 63–78. · doi:10.1007/978-3-319-58640-3_6
  29. Avidan, A. and C. Weissman, Record completeness and data concordance in an anesthesia information management system using context-sensitive mandatory data-entry fields. International Journal of Medical Informatics, 2012. 81(3): p. 173–181. · DOI
  30. McCormack, J.L. and J.S. Ash, Clinician perspectives on the quality of patient data used for clinical decision support: a qualitative study AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2012. 2012(Table 1): p. 1302–1309.
  31. Roomaney, R.A., et al., Availability and quality of routine morbidity data: Review of studies in South Africa. Journal of the American Medical Informatics Association, 2017. 24(e1): p. e194-e206. · DOI
  32. Zhang, Y. and G. Koru, Understanding and detecting defects in healthcare administration data: Toward higher data quality to better support healthcare operations and decisions. Journal of the American Medical Informatics Association, 2020. 27(3): p. 386–395. · DOI
  33. Feder, S.L., Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res, 2018. 40(5): p. 753–766. · DOI · Paper page
  34. Reimer, A.P., A. Milinovich, and E.A. Madigan, Data quality assessment framework to assess electronic medical record data for use in research. International Journal of Medical Informatics, 2016. 90: p. 40–47. · DOI · Paper page
  35. Yadav, S., et al., Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. Journal of the American Medical Informatics Association, 2017. 24(1): p. 140–144. · DOI · Paper page
  36. Abiy, R., et al., A Comparison of Electronic Medical Record Data to Paper Records in Antiretroviral Therapy Clinic in Ethiopia: What is affecting the Quality of the Data? Online J Public Health Inform, 2018. 10(2): p. e212-e212. · DOI · Paper page
  37. Daymont, C., et al., Automated identification of implausible values in growth data from pediatric electronic health records. J Am Med Inform Assoc, 2017. 24(6): p. 1080–1087. · DOI
  38. Callahan, T.J., et al., A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks. EGEMS (Wash DC), 2017. 5(1): p. 8.
  39. A Data Quality Ontology for the Secondary Use of EHR Data AMIA … Annual Symposium proceedings. AMIA Symposium, 2015. 2015: p. 1937–1946.
  40. Wang, Z., et al., Rule-Based Data Quality Assessment and Monitoring System in Healthcare Facilities. Stud Health Technol Inform, 2019. 257: p. 460–467.
  41. Pipino, L.L., Y.W. Lee, and R.Y. Wang, Data Quality Assessment Communications of the ACM, 2002. · doi:10.1145/505999.506010
  42. Woodall, P., M. Oberhofer, and A. Borek, A classification of data quality assessment and improvement methods. International Journal of Information Quality, 2014. 3(4): p. 298–321. · DOI · Paper page
  43. Pageler, N.M., et al., A rational approach to legacy data validation when transitioning between electronic health record systems. Journal of the American Medical Informatics Association, 2016. 23(5): p. 991–994. · DOI
  44. Kahn, M.G., et al., Transparent reporting of data quality in distributed data networks. EGEMS (Wash DC), 2015. 3(1): p. 1052.
  45. Hartzema, A.G., et al., Managing data quality for a drug safety surveillance system. Drug Safety, 2013. 36(SUPPL.1): p. S49-S58.
  46. Terry, A.L., et al., A basic model for assessing primary health care electronic medical record data quality. BMC Medical Informatics and Decision Making, 2019. 19(1). · DOI · Paper page
  47. Rogers, J.R., et al., A Data Element-Function Conceptual Model for Data Quality Checks. EGEMS (Wash DC), 2019. 7(1): p. 17.
  48. Callahan, T., et al., Reporting Data Quality Assessment Results: Identifying Individual and Organizational Barriers and Solutions. EGEMS (Wash DC), 2017. 5(1): p. 16.
  49. Data Quality and Interoperability Challenges for eHealth Exchange Participants: Observations from the Department of Veterans Affairs’ Virtual Lifetime Electronic Record Health Pilot Phase AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 307–314.
  50. Estiri, H., et al., A federated EHR network data completeness tracking system. Journal of the American Medical Informatics Association, 2019. 26(7): p. 637–645. · DOI
  51. Hart, R. and M.H. Kuo, Better Data Quality for Better Healthcare Research Results - A Case Study. Stud Health Technol Inform, 2017. 234: p. 161–166.
  52. Huser, V., et al., Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Data Sets. EGEMS (Wash DC), 2016. 4(1): p. 1239.
  53. Juárez, D., et al., A Generic Method and Implementation to Evaluate and Improve Data Quality in Distributed Research Networks. Methods of Information in Medicine, 2019. 58(2–3): p. 86–93.
  54. Kapsner, L.A., et al., Moving Towards an EHR Data Quality Framework: The MIRACUM Approach. Stud Health Technol Inform, 2019. 267: p. 247–253.
  55. Khare, R., et al., Design and Refinement of a Data Quality Assessment Workflow for a Large Pediatric Research Network. EGEMS (Wash DC), 2019. 7(1): p. 36.
  56. Lack, D., et al., Early detection of potential errors during patient treatment planning. Journal of Applied Clinical Medical Physics, 2018. 19(5): p. 724–732. · doi:10.1002/acm2.12388
  57. Monda, J., J. Keipeer, and M.C. Were, Data integrity module for data quality assurance within an e-health system in sub-Saharan Africa. Telemed J E Health, 2012. 18(1): p. 5–10. · doi:10.1089/tmj.2010.0224
  58. Rabia, L., I.A. Amarouche, and K. Beghdad Bey. Rule-based approach for detecting dirty data in discharge summaries. 2018. · DOI
  59. Improving Clinical Data Integrity by using Data Adjudication Techniques for Data Received through a Health Information Exchange (HIE) AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 1894–1901.
  60. van der Bij, S., et al., Improving the quality of EHR recording in primary care: A data quality feedback tool. Journal of the American Medical Informatics Association, 2017. 24(1): p. 81–87. · DOI
  61. Walker, K.L., et al., Using the CER Hub to ensure data quality in a multi-institution smoking cessation study. Journal of the American Medical Informatics Association, 2014. 21(6): p. 1129–1135. · DOI
  62. Tute, E., I. Scheffner, and M. Marschollek, A method for interoperable knowledge-based data quality assessment. BMC Medical Informatics and Decision Making, 2021. 21(1). · DOI
  63. Estiri, H., et al., Exploring completeness in clinical data research networks with DQe-c. Journal of the American Medical Informatics Association, 2018. 25(1): p. 17–24. · DOI
  64. Nasir, A., V. Gurupur, and X. Liu, A new paradigm to analyze data completeness of patient data. Applied Clinical Informatics, 2016. 7(3): p. 745–764. · DOI
  65. Silva, J.M., et al., A community-driven validation service for standard medical imaging objects Computer Standards and Interfaces, 2019. 61(May 2018): p. 121–128. · doi:10.1016/j.csi.2018.06.003
  66. Vanbrabant, L., et al., Quality of input data in emergency department simulations: Framework and assessment techniques Simulation Modelling Practice and Theory, 2019. 91(December 2018): p. 83–101. · DOI
  67. Álvarez Sánchez, R., et al., TAQIH, a tool for tabular data quality assessment and improvement in the context of health data. Computer Methods and Programs in Biomedicine, 2019. 181: 104824. · DOI
  68. Noselli, M., et al. MonAT: A visual web-based tool to profile health data quality. 2017. · DOI
  69. Yoon, D., et al., Conversion and data quality assessment of electronic health record data at a korean tertiary teaching hospital to a common data model for distributed network research. Healthcare Informatics Research, 2016. 22(1): p. 54–58. · DOI
  70. Dziadkowiec, O., et al., Using a Data Quality Framework to Clean Data Extracted from the Electronic Health Record: A Case Study. EGEMS (Wash DC), 2016. 4(1): p. 1201.
  71. A Framework for Data Quality Assessment in Clinical Research Datasets AMIA … Annual Symposium proceedings. AMIA Symposium, 2017. 2017: p. 1080–1089.
  72. Daniel, C., et al., Initializing a hospital-wide data quality program. The AP-HP experience Computer Methods and Programs in Biomedicine, 2019. 181: 104804. · DOI · Paper page
  73. Welch, G., et al., Data Cleaning in the Evaluation of a Multi-Site Intervention Project. EGEMS (Wash DC), 2017. 5(3): p. 4–4.
  74. Huser, V., et al., Extending Achilles Heel Data Quality Tool with New Rules Informed by Multi-Site Data Quality Comparison. Stud Health Technol Inform, 2019. 264: p. 1488–1489.
  75. Liao, T.V., et al., Evaluation of medication errors with implementation of electronic health record technology in the medical intensive care unit. Open Access Journal of Clinical Trials, 2017. 9: p. 31–40. · DOI
  76. Concordance of Electronic Health Record (EHR) Data Describing Delirium at a VA Hospital AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 1066–1071.
  77. Estiri, H. and S.N. Murphy, Semi-supervised encoding for outlier detection in clinical observation data. Comput Methods Programs Biomed, 2019. · doi:10.1101/334771
  78. Estiri, H., J.G. Klann, and S.N. Murphy, A clustering approach for detecting implausible observation values in electronic health records data. BMC Med Inform Decis Mak, 2019. 19(1): p. 142. · DOI
  79. Ling, Y., et al. An error detecting and tagging framework for reducing data entry errors in electronic medical records (EMR) system. 2013. · DOI
  80. Lai, K.H., et al., Automated misspelling detection and correction in clinical free-text records. Journal of Biomedical Informatics, 2015. 55: p. 188–195. · DOI · Paper page
  81. Peng, M., et al., Exploration of association rule mining for coding consistency and completeness assessment in inpatient administrative health data. Journal of Biomedical Informatics, 2018. 79(February): p. 41–47. · DOI
  82. Wang, Z., M. Penning, and M. Zozus, Analysis of Anesthesia Screens for Rule-Based Data Quality Assessment Opportunities. Stud Health Technol Inform, 2019. 257: p. 473–478.
  83. Johnson, S.G., et al., Application of an ontology for characterizing data quality for a secondary use of EHR data. Applied Clinical Informatics, 2016. 7(1): p. 69–88. · DOI · Paper page
  84. Boselli, R., et al., A policy-based cleansing and integration framework for labour and healthcare data. 2014. p. 141–168. · DOI
  85. Ferrao, J.C., et al., Preprocessing structured clinical data for predictive modeling and decision support. A roadmap to tackle the challenges. Appl Clin Inform, 2016. 7(4): p. 1135–1153. · DOI

Cited by

See all 69

Metadata

Authors
Ozonze, O., Scott, P. J., & Hopgood, A. A.
Venue
Journal of Medical Systems
Published
2023-02-13
Volume
47
Issue
1
Pages
23-23
ISSN
  • 0148-5598 (ISSN-L)
  • 1573-689X
OpenAlex
W4320482217
Cited by
69
References
15
FWCI
14.47
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
University of Portsmouth (University of Portsmouth)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automating Electronic Health Record Data Quality Assessment. https://app.cochat.ai/papers/10.1007/s10916-022-01892-2
The paper
Ozonze, O., Scott, P. J., & Hopgood, A. A. (2023). Automating Electronic Health Record Data Quality Assessment. Journal of Medical Systems. https://doi.org/10.1007/s10916-022-01892-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.