CoChat

Automating Electronic Health Record Data Quality Assessment

Open access
Ozonze, Obinwa, Scott, Philip J., Hopgood, Adrian A. · 2023 · Journal of Medical Systems
Cited by 69

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Information systems such as Electronic Health Record (EHR) systems are susceptible to data quality (DQ) issues. Given the growing importance of EHR data, there is an increasing demand for strategies and tools to help ensure that available data are fit for use. However, developing reliable data quality assessment (DQA) tools necessary for guiding and evaluating improvement efforts has remained a fundamental challenge. This review examines the state of research on operationalising EHR DQA, mainly automated tooling, and highlights necessary considerations for future implementations. We reviewed 1841 articles from PubMed, Web of Science, and Scopus published between 2011 and 2021. 23 DQA programs deployed in real-world settings to assess EHR data quality (n = 14), and a few experimental prototypes (n = 9), were identified. Many of these programs investigate completeness (n = 15) and value conformance (n = 12) quality dimensions and are backed by knowledge items gathered from domain experts (n = 9), literature reviews and existing DQ measurements (n = 3). A few DQA programs also explore the feasibility of using data-driven techniques to assess EHR data quality automatically. Overall, the automation of EHR DQA is gaining traction, but current efforts are fragmented and not backed by relevant theory. Existing programs also vary in scope, type of data supported, and how measurements are sourced. There is a need to standardise programs for assessing EHR data quality, as current evidence suggests their quality may be unknown.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10916-022-01892-2

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 83

  1. Warren, L.R., et al., Improving data sharing between acute hospitals in England: an overview of health record system distribution and retrospective observational analysis of inter-hospital transitions of care. BMJ Open, 2019. 9(12): p. e031637. · DOI · Paper page
  2. Atasoy, H., B.N. Greenwood, and J.S. McCullough, The Digitization of Patient Care: A Review of the Effects of Electronic Health Records on Health Care Quality and Utilization. Annu Rev Public Health, 2019. 40: p. 487–500. · DOI · Paper page
  3. Hayrinen, K., K. Saranto, and P. Nykanen, Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform, 2008. 77(5): p. 291–304. · DOI · Paper page
  4. Topol, E., The Topol Review Preparing the Healthcare Workforce to Deliver the Digital Future, 2019: p. 1–48.
  5. Safran, C., Update on Data Reuse in Health Care. Yearb Med Inform, 2017. 26(1): p. 24–27. · doi:10.15265/iy-2017-013
  6. Vuokko, R., et al., Secondary Use of Structured Patient Data: Interim Results of A Systematic Review. 2015. p. 291–295.
  7. Bayley, K.B., et al., Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med Care, 2013. 51(8 Suppl 3): p. S80-6.
  8. WHO, Administrative Errors: Technical Series on Safer Primary Care, in WHO Press. 2016: Geneva.
  9. Ghosh, A., S. McCarthy, and E. Halcomb, Perceptions of primary care staff on a regional data quality intervention in Australian general practice: A qualitative study. BMC Family Practice, 2016. 17(1). · DOI
  10. Salomon, R.M., et al., Openness of patients’ reporting with use of electronic records: Psychiatric clinicians’ views. Journal of the American Medical Informatics Association, 2010. 17(1): p. 54–60. · DOI · Paper page
  11. Peivandi, S., et al., Evaluation and comparison of errors on nursing notes created by online and offline speech recognition technology and handwritten: an interventional study. BMC Medical Informatics and Decision Making, 2022. 22(1): p. 96. · DOI
  12. Colin, N.V., et al., Understanding the Impact of Variations in Measurement Period Reporting for Electronic Clinical Quality Measures. EGEMS (Wash DC), 2018. 6(1): p. 17.
  13. Bowman, S., Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag, 2013. 10: p. 1c-1c.
  14. O’Donnell, H.C., et al., Physicians’ Attitudes Towards Copy and Pasting in Electronic Note Writing. Journal of General Internal Medicine, 2009. 24(1): p. 63–68. · DOI
  15. Coleman, N., et al., From patient care to research: A validation study examining the factors contributing to data quality in a primary care electronic medical record database. BMC Family Practice, 2015. 16(1). · DOI
  16. Economic analysis of the prevalence and clinical and economic burden of medication error in England BMJ Quality & Safety, 2021. 30(2): p. 96. · DOI · Paper page
  17. Zozus, M.N., M. Penning, and W.E. Hammond, Factors impacting physician use of information charted by others. JAMIA Open, 2019. 2(1): p. 107–114. · DOI
  18. Munyisia, E.N., D. Reid, and P. Yu, Accuracy of outpatient service data for activity-based funding in New South Wales, Australia. Health Inf Manag, 2017. 46(2): p. 78–86.
  19. Kaplan, B., How Should Health Data Be Used?: Privacy, Secondary Use, and Big Data Sales. Cambridge Quarterly of Healthcare Ethics, 2016. 25(2): p. 312–329. · DOI · Paper page
  20. Nouraei, S.A.R., et al., Accuracy of clinician-clinical coder information handover following acute medical admissions: Implication for using administrative datasets in clinical outcomes management. Journal of Public Health (United Kingdom), 2016. 38(2): p. 352–362.
  21. Feldman, K., et al., Beyond volume: The impact of complex healthcare data on the machine learning pipeline Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017. 10344 LNAI: p. 150–169. · DOI
  22. Hanauer, D.A., et al., Complexities, variations, and errors of numbering within clinical notes: The potential impact on information extraction and cohort-identification. BMC Medical Informatics and Decision Making, 2019. 19(Suppl 3): p. 75–75. · DOI
  23. Batini, C., et al., Methodologies for data quality assessment and improvement. ACM computing surveys (CSUR), 2009. 41(3): p. 16. · DOI · Paper page
  24. Veiga, A.K., et al., A conceptual framework for quality assessment and management of biodiversity data. PLoS ONE, 2017. 12(6): p. e0178731-e0178731. · DOI
  25. Kelly, M.M., R.J. Coller, and P.L. Hoonakker, Inpatient Portals for Hospitalized Patients and Caregivers: A Systematic Review. J Hosp Med, 2018. 13(6): p. 405–412. · DOI
  26. Isaksen, H., et al., Design of tooltips for data fields: A field experiment of logging use of tooltips and data correctness. 2017. p. 63–78. · DOI
  27. Avidan, A. and C. Weissman, Record completeness and data concordance in an anesthesia information management system using context-sensitive mandatory data-entry fields. International Journal of Medical Informatics, 2012. 81(3): p. 173–181. · DOI
  28. McCormack, J.L. and J.S. Ash, Clinician perspectives on the quality of patient data used for clinical decision support: a qualitative study AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2012. 2012(Table 1): p. 1302–1309.
  29. Roomaney, R.A., et al., Availability and quality of routine morbidity data: Review of studies in South Africa. Journal of the American Medical Informatics Association, 2017. 24(e1): p. e194-e206. · DOI
  30. Zhang, Y. and G. Koru, Understanding and detecting defects in healthcare administration data: Toward higher data quality to better support healthcare operations and decisions. Journal of the American Medical Informatics Association, 2020. 27(3): p. 386–395. · DOI
  31. Feder, S.L., Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res, 2018. 40(5): p. 753–766. · DOI · Paper page
  32. Reimer, A.P., A. Milinovich, and E.A. Madigan, Data quality assessment framework to assess electronic medical record data for use in research. International Journal of Medical Informatics, 2016. 90: p. 40–47. · DOI · Paper page
  33. Yadav, S., et al., Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. Journal of the American Medical Informatics Association, 2017. 24(1): p. 140–144. · DOI · Paper page
  34. Abiy, R., et al., A Comparison of Electronic Medical Record Data to Paper Records in Antiretroviral Therapy Clinic in Ethiopia: What is affecting the Quality of the Data? Online J Public Health Inform, 2018. 10(2): p. e212-e212. · DOI · Paper page
  35. Daymont, C., et al., Automated identification of implausible values in growth data from pediatric electronic health records. J Am Med Inform Assoc, 2017. 24(6): p. 1080–1087. · DOI
  36. Callahan, T.J., et al., A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks. EGEMS (Wash DC), 2017. 5(1): p. 8.
  37. A Data Quality Ontology for the Secondary Use of EHR Data AMIA … Annual Symposium proceedings. AMIA Symposium, 2015. 2015: p. 1937–1946.
  38. Wang, Z., et al., Rule-Based Data Quality Assessment and Monitoring System in Healthcare Facilities. Stud Health Technol Inform, 2019. 257: p. 460–467.
  39. Pipino, L.L., Y.W. Lee, and R.Y. Wang, Data Quality Assessment Communications of the ACM, 2002. · doi:10.1145/505999.506010
  40. Woodall, P., M. Oberhofer, and A. Borek, A classification of data quality assessment and improvement methods. International Journal of Information Quality, 2014. 3(4): p. 298–321. · DOI · Paper page
  41. Pageler, N.M., et al., A rational approach to legacy data validation when transitioning between electronic health record systems. Journal of the American Medical Informatics Association, 2016. 23(5): p. 991–994. · DOI
  42. Kahn, M.G., et al., Transparent reporting of data quality in distributed data networks. EGEMS (Wash DC), 2015. 3(1): p. 1052.
  43. Hartzema, A.G., et al., Managing data quality for a drug safety surveillance system. Drug Safety, 2013. 36(SUPPL.1): p. S49-S58.
  44. Terry, A.L., et al., A basic model for assessing primary health care electronic medical record data quality. BMC Medical Informatics and Decision Making, 2019. 19(1). · DOI · Paper page
  45. Rogers, J.R., et al., A Data Element-Function Conceptual Model for Data Quality Checks. EGEMS (Wash DC), 2019. 7(1): p. 17.
  46. Callahan, T., et al., Reporting Data Quality Assessment Results: Identifying Individual and Organizational Barriers and Solutions. EGEMS (Wash DC), 2017. 5(1): p. 16.
  47. Data Quality and Interoperability Challenges for eHealth Exchange Participants: Observations from the Department of Veterans Affairs’ Virtual Lifetime Electronic Record Health Pilot Phase AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 307–314.
  48. Estiri, H., et al., A federated EHR network data completeness tracking system. Journal of the American Medical Informatics Association, 2019. 26(7): p. 637–645. · DOI
  49. Hart, R. and M.H. Kuo, Better Data Quality for Better Healthcare Research Results - A Case Study. Stud Health Technol Inform, 2017. 234: p. 161–166.
  50. Huser, V., et al., Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Data Sets. EGEMS (Wash DC), 2016. 4(1): p. 1239.
  51. Juárez, D., et al., A Generic Method and Implementation to Evaluate and Improve Data Quality in Distributed Research Networks. Methods of Information in Medicine, 2019. 58(2–3): p. 86–93.
  52. Kapsner, L.A., et al., Moving Towards an EHR Data Quality Framework: The MIRACUM Approach. Stud Health Technol Inform, 2019. 267: p. 247–253.
  53. Khare, R., et al., Design and Refinement of a Data Quality Assessment Workflow for a Large Pediatric Research Network. EGEMS (Wash DC), 2019. 7(1): p. 36.
  54. Lack, D., et al., Early detection of potential errors during patient treatment planning. Journal of Applied Clinical Medical Physics, 2018. 19(5): p. 724–732. · doi:10.1002/acm2.12388
  55. Monda, J., J. Keipeer, and M.C. Were, Data integrity module for data quality assurance within an e-health system in sub-Saharan Africa. Telemed J E Health, 2012. 18(1): p. 5–10. · DOI
  56. Rabia, L., I.A. Amarouche, and K. Beghdad Bey. Rule-based approach for detecting dirty data in discharge summaries. 2018. · DOI
  57. Improving Clinical Data Integrity by using Data Adjudication Techniques for Data Received through a Health Information Exchange (HIE) AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 1894–1901.
  58. van der Bij, S., et al., Improving the quality of EHR recording in primary care: A data quality feedback tool. Journal of the American Medical Informatics Association, 2017. 24(1): p. 81–87. · DOI
  59. Walker, K.L., et al., Using the CER Hub to ensure data quality in a multi-institution smoking cessation study. Journal of the American Medical Informatics Association, 2014. 21(6): p. 1129–1135. · DOI
  60. Tute, E., I. Scheffner, and M. Marschollek, A method for interoperable knowledge-based data quality assessment. BMC Medical Informatics and Decision Making, 2021. 21(1). · DOI
  61. Estiri, H., et al., Exploring completeness in clinical data research networks with DQe-c. Journal of the American Medical Informatics Association, 2018. 25(1): p. 17–24. · DOI
  62. Nasir, A., V. Gurupur, and X. Liu, A new paradigm to analyze data completeness of patient data. Applied Clinical Informatics, 2016. 7(3): p. 745–764. · DOI
  63. Silva, J.M., et al., A community-driven validation service for standard medical imaging objects Computer Standards and Interfaces, 2019. 61(May 2018): p. 121–128. · doi:10.1016/j.csi.2018.06.003
  64. Vanbrabant, L., et al., Quality of input data in emergency department simulations: Framework and assessment techniques Simulation Modelling Practice and Theory, 2019. 91(December 2018): p. 83–101. · DOI
  65. Álvarez Sánchez, R., et al., TAQIH, a tool for tabular data quality assessment and improvement in the context of health data. Computer Methods and Programs in Biomedicine, 2019. 181: 104824. · DOI
  66. Noselli, M., et al. MonAT: A visual web-based tool to profile health data quality. 2017. · DOI
  67. Yoon, D., et al., Conversion and data quality assessment of electronic health record data at a korean tertiary teaching hospital to a common data model for distributed network research. Healthcare Informatics Research, 2016. 22(1): p. 54–58. · DOI
  68. Dziadkowiec, O., et al., Using a Data Quality Framework to Clean Data Extracted from the Electronic Health Record: A Case Study. EGEMS (Wash DC), 2016. 4(1): p. 1201.
  69. A Framework for Data Quality Assessment in Clinical Research Datasets AMIA … Annual Symposium proceedings. AMIA Symposium, 2017. 2017: p. 1080–1089.
  70. Daniel, C., et al., Initializing a hospital-wide data quality program. The AP-HP experience Computer Methods and Programs in Biomedicine, 2019. 181: 104804. · DOI · Paper page
  71. Welch, G., et al., Data Cleaning in the Evaluation of a Multi-Site Intervention Project. EGEMS (Wash DC), 2017. 5(3): p. 4–4.
  72. Huser, V., et al., Extending Achilles Heel Data Quality Tool with New Rules Informed by Multi-Site Data Quality Comparison. Stud Health Technol Inform, 2019. 264: p. 1488–1489.
  73. Liao, T.V., et al., Evaluation of medication errors with implementation of electronic health record technology in the medical intensive care unit. Open Access Journal of Clinical Trials, 2017. 9: p. 31–40. · DOI
  74. Concordance of Electronic Health Record (EHR) Data Describing Delirium at a VA Hospital AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2014. 2014: p. 1066–1071.
  75. Estiri, H. and S.N. Murphy, Semi-supervised encoding for outlier detection in clinical observation data. Comput Methods Programs Biomed, 2019. · doi:10.1101/334771
  76. Estiri, H., J.G. Klann, and S.N. Murphy, A clustering approach for detecting implausible observation values in electronic health records data. BMC Med Inform Decis Mak, 2019. 19(1): p. 142. · DOI
  77. Ling, Y., et al. An error detecting and tagging framework for reducing data entry errors in electronic medical records (EMR) system. 2013. · DOI
  78. Lai, K.H., et al., Automated misspelling detection and correction in clinical free-text records. Journal of Biomedical Informatics, 2015. 55: p. 188–195. · DOI · Paper page
  79. Peng, M., et al., Exploration of association rule mining for coding consistency and completeness assessment in inpatient administrative health data. Journal of Biomedical Informatics, 2018. 79(February): p. 41–47. · DOI
  80. Wang, Z., M. Penning, and M. Zozus, Analysis of Anesthesia Screens for Rule-Based Data Quality Assessment Opportunities. Stud Health Technol Inform, 2019. 257: p. 473–478.
  81. Johnson, S.G., et al., Application of an ontology for characterizing data quality for a secondary use of EHR data. Applied Clinical Informatics, 2016. 7(1): p. 69–88. · DOI · Paper page
  82. Boselli, R., et al., A policy-based cleansing and integration framework for labour and healthcare data. 2014. p. 141–168. · DOI
  83. Ferrao, J.C., et al., Preprocessing structured clinical data for predictive modeling and decision support. A roadmap to tackle the challenges. Appl Clin Inform, 2016. 7(4): p. 1135–1153. · DOI

Cited by

Metadata

Authors
Ozonze, O., Scott, P. J., & Hopgood, A. A.
Venue
Journal of Medical Systems
Published
2023-02-13
Volume
47
Issue
1
Pages
23-23
ISSN
  • 0148-5598 (ISSN-L)
  • 1573-689X
OpenAlex
W4320482217
Cited by
69
References
17
FWCI
14.47
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
University of Portsmouth (University of Portsmouth)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.