Cited by 38Medical Coding and Health InformationElectronic Health Records SystemsData Quality and Management

Data consistency in the English Hospital Episodes Statistics database

Open access
Hardy, Flavien, Heyl, Johannes, Tucker, Katie, Hopper, Adrian, Marchã, Maria J, Briggs, Tim W R, Yates, Jeremy, Day, Jamie +3 more·2022·BMJ Health & Care Informatics

Abstract

Background: To gain maximum insight from large administrative healthcare datasets it is important to understand their data quality. Although a gold standard against which to assess criterion validity rarely exists for such datasets, internal consistency can be evaluated. We aimed to identify inconsistencies in the recording of mandatory International Statistical Classification of Diseases and Related Health Problems, tenth revision (ICD-10) codes within the Hospital Episodes Statistics dataset in England. Methods: Three exemplar medical conditions where recording is mandatory once diagnosed were chosen: autism, type II diabetes mellitus and Parkinson’s disease dementia. We identified the first occurrence of the condition ICD-10 code for a patient during the period April 2013 to March 2021 and in subsequent hospital spells. We designed and trained random forest classifiers to identify variables strongly associated with recording inconsistencies. Results: For autism, diabetes and Parkinson’s disease dementia respectively, 43.7%, 8.6% and 31.2% of subsequent spells had inconsistencies. Coding inconsistencies were highly correlated with non-coding of an underlying condition, a change in hospital trust and greater time between the spell with the first coded diagnosis and the subsequent spell. For patients with diabetes or Parkinson’s disease dementia, the code recording for spells without an overnight stay were found to have a higher rate of inconsistencies. Conclusions: Data inconsistencies are relatively common for the three conditions considered. Where these mandatory diagnoses are not recorded in administrative datasets, and where clinical decisions are made based on such data, there is potential for this to impact patient care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjhci-2022-100633

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Structured datasets can provide a narrow view of a patient's journey because about 80% of patient data are found in free-text format.

    “However, structured datasets are not always available and, even when they are, they can provide a narrow view of a patient's journey, because about 80% of patient data are found in freetext format.1,2”

    From Foresight-a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study (2024) · cites this paper as [2]

    The Introduction (page not available in our copy) says administrative healthcare data often lack clinical details, supporting the point that a structured dataset can give a limited view. Neither the Introduction nor the Methods’ description of coding from clinical notes and discharge summaries gives the claimed figure that about 80% of patient data are in free text; searches for that figure and for free-text data found no supporting passage. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • About 80% of patient data are found in free-text format.

    “because about 80% of patient data are found in freetext format.1,2”

    From Foresight-a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study (2024) · cites this paper as [2]

    The Methods section (Study design and data collection; page unavailable) says HES data are taken from clinical notes and discharge summaries and entered by clinical coders. It gives no estimate of the proportion of patient data in free-text format. Searches for the 80% figure and for the share of unstructured patient records found no supporting passage. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [2]Griffith GJ, Morris TT, Tudball MJ, et al. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat Commun 2020; 11: 1– 12. 10.1038/s41467-020-19478-2 33184277 PMC7665028 · DOI · Paper page
  2. [3]Stulberg JJ, Haut ER. Practical guide to surgical data sets: healthcare cost and utilization project national inpatient sample (NIS). JAMA Surg 2018; 153: 586– 7. 10.1001/jamasurg.2018.0542 29617533 · DOI
  3. [4]Benchimol EI, Smeeth L, Guttmann A, et al. The reporting of studies conducted using observational Routinely-collected health data (record) statement. PLoS Med 2015; 12: e1001885. 10.1371/journal.pmed.1001885 26440803 PMC4595218 · DOI · Paper page
  4. [6]NHS Digital. National clinical coding standards ICD-10Accurate data for quality information. 5th Edition. Leeds, UK: Terminology and Classifications Delivery Service, 2021.
  5. [7]Herbert A, Wijlaars L, Zylbersztejn A, et al. Data resource profile: Hospital episode statistics admitted patient care (Hes APC). Int J Epidemiol 2017; 46: 1093– 1093i. 10.1093/ije/dyx015 28338941 PMC5837677 · DOI · Paper page
  6. [9]Sundararajan V, Henderson T, Perry C, et al. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol 2004; 57: 1288– 94. 10.1016/j.jclinepi.2004.03.012 15617955 · DOI · Paper page
  7. [10]Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic Hospital records: an observational study. Lancet 2018; 391: 1775– 82. 10.1016/S0140-6736(18)30668-8 29706364 PMC5946808 · DOI
  8. [11]Soong JTY, Kaubryte J, Liew D, et al. Dr foster global frailty score: an international retrospective observational study developing and validating a risk prediction model for hospitalised older persons from administrative data sets. BMJ Open 2019; 9: e026759. 10.1136/bmjopen-2018-026759 PMC6596946 31230009 · DOI
  9. [13]Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in python. Journal of machine Learning research 2011; 12: 2825– 30.
  10. [14]Lundberg SM, Lee S-I. A unified approach to interpreting model predictions. Proceedings of the 31st international conference on neural information processing systems 2017; 2017: 4768– 77.
  11. [15]Lundberg SM, Erion GG, Lee S-I. Consistent individualized feature Attribution for tree ensembles. arXiv preprint arXiv 2018: 180203888.
  12. [16]Flach P, Kull M. Precision-recall-gain curves: PR analysis done right. Adv Neural Inf Process Syst 2015; 28.
  13. [17]DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837– 45. 10.2307/2531595 3203132 · DOI · Paper page
  14. [18]Sun X, Xu W. Fast implementation of DeLong’s algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Process Lett 2014; 21: 1389– 93. 10.1109/LSP.2014.2337313 · DOI
  15. [19]Strobl C, Boulesteix A-L, Zeileis A, et al. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 2007; 8: 1– 21. 10.1186/1471-2105-8-25 17254353 PMC1796903 · DOI
  16. [20]Strobl C, Boulesteix A-L, Kneib T, et al. Conditional variable importance for random forests. BMC Bioinformatics 2008; 9: 1– 11. 10.1186/1471-2105-9-307 18620558 PMC2491635 · DOI
  17. [21]Gray WK, Day J, Horrocks M. Editor's Choice - Volume-Outcome Relationships in Elective Abdominal Aortic Aneurysm Surgery: Analysis of the UK Hospital Episodes Statistics Database for the Getting It Right First Time (GIRFT) Programme. Eur J Vasc Endovasc Surg 2020; 60: 509– 17. 10.1016/j.ejvs.2020.07.069 32807679 · DOI
  18. [22]Nouraei SAR, Mace AD, Middleton SE, et al. A stratified analysis of the perioperative outcome of 17623 patients undergoing major head and neck cancer surgery in England over 10 years: towards an Informatics-based outcomes surveillance framework. Clin Otolaryngol 2017; 42: 11– 28. 10.1111/coa.12649 26990866 · DOI
  19. [23]Navaratnam AV, Gray WK, Day J, et al. Patient factors and temporal trends associated with COVID-19 in-hospital mortality in England: an observational study using administrative data. Lancet Respir Med 2021; 9: 397– 406. 10.1016/S2213-2600(20)30579-8 33600777 PMC7906650 · DOI
  20. [24]Gray WK, Navaratnam AV, Day J, et al. Changes in COVID-19 in-hospital mortality in hospitalised adults in England over the first seven months of the pandemic: an observational study using administrative data. Lancet Reg Health Eur 2021; 5: 100104. 10.1016/j.lanepe.2021.100104 33969337 PMC8086562 · DOI
  21. [27]Gray WK, Takhar AS, Navaratnam AV, et al. Safety of day-case paediatric tonsillectomy in England: an analysis of administrative data for the getting it right first time programme. Anaesthesia 2022; 77: 277– 85. 10.1111/anae.15562 34530496 · DOI
  22. [28]Jewell A, Broadbent M, Hayes RD, et al. Impact of matching error on linked mortality outcome in a data linkage of secondary mental health data with Hospital episode statistics (Hes) and mortality records in South East London: a cross-sectional study. BMJ Open 2020; 10: e035884. 10.1136/bmjopen-2019-035884 PMC7342822 32641360 · DOI
  23. [29]Reddy S, Fox J, Purohit MP. Artificial intelligence-enabled healthcare delivery. J R Soc Med 2019; 112: 22– 8. 10.1177/0141076818815510 30507284 PMC6348559 · DOI · Paper page
  24. [30]Panch T, Mattie H, Atun R. Artificial intelligence and algorithmic bias: implications for health systems. J Glob Health 2019; 9: 010318. 10.7189/jogh.09.020318 PMC6875681 31788229 · DOI · Paper page

Cited by

Sorted by citations

Authors
Hardy, F., Heyl, J., Tucker, K., Hopper, A., Marchã, M. J., Briggs, T. W. R., Yates, J., Day, J., Wheeler, A., Eve-Jones, S., & Gray, W. K.
Affiliations
  • NHS England (GB)
  • Royal Free London NHS Foundation Trust (GB)
  • Guy's and St Thomas' NHS Foundation Trust (GB)
  • Science and Technology Facilities Council (GB)
  • Royal National Orthopaedic Hospital NHS Trust (GB)
Corresponding author
Hardy, Flavien
Venue
BMJ Health & Care Informatics
Publisher
BMJ
Published
2022-10
Volume
29
Issue
1
Pages
e100633
ISSN
2632-1009 (ISSN-L)
OpenAlex
W4307972173
Cited by
38
References
1
Citations by year
2026: 9 · 2025: 11 · 2024: 10 · 2023: 8
FWCI
5.44
Type
journal article
Language
en
Open access
Locations
  • BMJ Health & Care Informatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • UCL Discovery (University College London) (submittedVersion)
Funders
Science and Technology Facilities Council (Fellowship from DiRAC, ST/S003916/1, ST/W002760/1)
Topics
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
Keywords
Hospital Episode Statistics, ICD-10 coding, data consistency, autism spectrum disorder, type 2 diabetes, Parkinson's disease dementia, hospital episodes, hospital trusts, overnight hospital stay, random forest classification
MeSH terms
Dementia, Diabetes Mellitus, Type 2, Hospitals, Humans, Parkinson Disease, International Classification of Diseases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review