Multisource and temporal variability in Portuguese hospital administrative datasets: Data quality implications
Open accessAbstract
BACKGROUND: Unexpected variability across healthcare datasets may indicate data quality issues and thereby affect the credibility of these data for reutilization. No gold-standard reference dataset or methods for variability assessment are usually available for these datasets. In this study, we aim to describe the process of discovering data quality implications by applying a set of methods for assessing variability between sources and over time in a large hospital database. METHODS: We described and applied a set of multisource and temporal variability assessment methods in a large Portuguese hospitalization database, in which variation in condition-specific hospitalization ratios derived from clinically coded data were assessed between hospitals (sources) and over time. We identified condition-specific admissions using the Clinical Classification Software (CCS), developed by the Agency of Health Care Research and Quality. A Statistical Process Control (SPC) approach based on funnel plots of condition-specific standardized hospitalization ratios (SHR) was used to assess multisource variability, whereas temporal heat maps and Information-Geometric Temporal (IGT) plots were used to assess temporal variability by displaying temporal abrupt changes in data distributions. Results were presented for the 15 most common inpatient conditions (CCS) in Portugal. MAIN FINDINGS: Funnel plot assessment allowed the detection of several outlying hospitals whose SHRs were much lower or higher than expected. Adjusting SHR for hospital characteristics, beyond age and sex, considerably affected the degree of multisource variability for most diseases. Overall, probability distributions changed over time for most diseases, although heterogeneously. Abrupt temporal changes in data distributions for acute myocardial infarction and congestive heart failure coincided with the periods comprising the transition to the International Classification of Diseases, 10th revision, Clinical Modification, whereas changes in the Diagnosis-Related Groups software seem to have driven changes in data distributions for both acute myocardial infarction and liveborn admissions. The analysis of heat maps also allowed the detection of several discontinuities at hospital level over time, in some cases also coinciding with the aforementioned factors. CONCLUSIONS: This paper described the successful application of a set of reproducible, generalizable and systematic methods for variability assessment, including visualization tools that can be useful for detecting abnormal patterns in healthcare data, also addressing some limitations of common approaches. The presented method for multisource variability assessment is based on SPC, which is an advantage considering the lack of gold standard for such process. Properly controlling for hospital characteristics and differences in case-mix for estimating SHR is critical for isolating data quality-related variability among data sources. The use of IGT plots provides an advantage over common methods for temporal variability assessment due its suitability for multitype and multimodal data, which are common characteristics of healthcare data. The novelty of this work is the use of a set of methods to discover new data quality insights in healthcare data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Variability in health data collected at the edge can impact the performance and generalizability of AI models.
“This variability can impact the performance and generalizability of AI models, making it essential to address data variability challenges.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [97]
Health data collected at the edge can be highly variable due to factors such as device sensors, user behavior, environmental conditions, and data collection protocols.
“Health data collected at the edge can be highly variable due to various factors such as device sensors, user behavior, environmental conditions, and data collection protocols [97].”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [97]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669 · see citing works
Time Series Analysis Forecasting and Control.
Bartholomew, D. J., Box, G. E. P., & Jenkins, G. M. (1971) · Operational Research Quarterly (1970-1977) · doi:10.2307/3008255 · Cited by 6,872 · see citing works
Big data in healthcare: management, analysis and future prospects
Dash, S., Shakyawar, S. K., Sharma, M., & Kaushik, S. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0217-0 · Cited by 1,747 · see citing works
Dataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553 · see citing works
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536 · see citing works
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · see citing works
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029 · see citing works
<b>Research Commentary</b> —Too Big to Fail: Large Samples and the <i>p</i> -Value Problem
Lin, M., Lucas, H. C., & Shmueli, G. (2013) · Information Systems Research · doi:10.1287/isre.2013.0480 · Cited by 972 · see citing works
Funnel plots for comparing institutional performance
Spiegelhalter, D. J. (2005) · Statistics in Medicine · doi:10.1002/sim.1970 · Cited by 832 · see citing works
The fickle P value generates irreproducible results
Halsey, L. G., Curran‐Everett, D., Vowler, S. L., & Drummond, G. B. (2015) · Nature Methods · doi:10.1038/nmeth.3288 · Cited by 753 · see citing works
Bray, F., & Parkin, D. M. (2008) · European Journal of Cancer · doi:10.1016/j.ejca.2008.11.032 · Cited by 634 · see citing works
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · see citing works
Statistical method from the viewpoint of quality control
Oppermann, R. H. (1940) · Journal of the Franklin Institute · doi:10.1016/s0016-0032(40)90985-1 · Cited by 271 · see citing works
Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012) · Medical Care · doi:10.1097/mlr.0b013e318257dd67 · Cited by 260 · see citing works
Handling over-dispersion of performance indicators
Spiegelhalter, D. J. (2005) · BMJ Quality & Safety · doi:10.1136/qshc.2005.013755 · Cited by 150 · see citing works
Statistical Methods for Healthcare Regulation: Rating, Screening and Surveillance
Spiegelhalter, D. J., Sherlaw‐Johnson, C., Bardsley, M., Blunt, I., Wood, C., & Grigg, O. A. J. (2011) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2011.01010.x · Cited by 147 · see citing works
SHRINE: Enabling Nationally Scalable Multi-Site Disease Studies
McMurry, A., Murphy, S. N., Macfadden, D. K., Weber, G. M., Simons, W. W., Orechia, J. A., Bickel, J. P., Wattanasin, N., Gilbert, C., Trevvett, P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0055811 · Cited by 125 · see citing works
Data quality assurance, monitoring, and reporting
Gassman, J., Owen, W. W., Kuntz, T. E., Martin, J. P., & Amoroso, W. P. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00095-k · Cited by 99 · see citing works
Asaria, P., Elliott, P., Douglass, M., Obermeyer, Z., Soljak, M., Majeed, A., & Ezzati, M. (2017) · The Lancet Public Health · doi:10.1016/s2468-2667(17)30032-4 · Cited by 94 · see citing works
Rothberg, M. B., Pekow, P. S., Priya, A., & Lindenauer, P. K. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1419 · Cited by 60 · see citing works
Funnel Plots and Their Emerging Application in Surgery
Mayer, E. K., Bottle, A., Rao, C., Darzi, A. W., & Athanasiou, T. (2009) · Annals of Surgery · doi:10.1097/sla.0b013e31819a47b1 · Cited by 55 · see citing works
Verburg, I. W. M., Holman, R., Peek, N. B., Abu–Hanna, A., & de Keizer, N. F. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280217700169 · Cited by 54 · see citing works
Doktorchik, C. T. A., Lu, M., Quan, H., Ringham, C., & Eastwood, C. A. (2019) · Health Information Management Journal · doi:10.1177/1833358319855031 · Cited by 49 · see citing works
The importance of clinical documentation improvement for Australian hospitals
Hay, P., Wilton, K., Barker, J., et al. (2019) · Health Information Management Journal · doi:10.1177/1833358319854185 · Cited by 46 · see citing works
Sáez, C., Zurriaga, Ó., Pérez‐Panadés, J., Melchor, I., Robles, M., & Garcia-Gomez, J. M. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw010 · Cited by 45 · see citing works
Self-citationUsing funnel plots in public health surveillance
Dover, D., & Schopflocher, D. (2011) · Population Health Metrics · doi:10.1186/1478-7954-9-58 · Cited by 43 · see citing works
Sáez, C., Robles, M., & Garcia-Gomez, J. M. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214545122 · Cited by 35 · see citing works
Self-citationSáez, C., & Garcia-Gomez, J. M. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.09.015 · Cited by 33 · see citing works
Self-citationEHRtemporalVariability: delineating temporal data-set shifts in electronic health records
Sáez, C., Gutiérrez‐Sacristán, A., Kohane, I. S., Garcia-Gomez, J. M., & Avillach, P. (2020) · GigaScience · doi:10.1093/gigascience/giaa079 · Cited by 32 · see citing works
Self-citationWright, A. T., Ash, J. S., Aaron, S., Ai, A., Hickman, T.-T. T., Wiesen, J. F., Galanter, W., McCoy, A. B., Schreiber, R., Longhurst, C., et al. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.08.001 · Cited by 31 · see citing works
Hospital heterogeneity: what drives the quality of health care
Ali, M. M., Salehnejad, R., & Mansur, M. (2017) · The European Journal of Health Economics · doi:10.1007/s10198-017-0891-9 · Cited by 31 · see citing works
Data Quality and Integration Issues in Electronic Health Records
Cruz-Correia, R. J., Rodrigues, P. P., Freitas, A., et al. (2009) · Chapman & Hall/CRC data mining and knowledge discovery series · doi:10.1201/9781420090413-c4 · Cited by 27 · see citing works
Self-citationSáez, C., Rodrigues, P. P., da Gama, J. M. P., et al. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0378-6 · Cited by 25 · see citing works
Self-citationAssareh, H., Achat, H. M., Stubbs, J. M., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147087 · Cited by 25 · see citing works
Statistical Quality Control in Clinical Trials
Svolba, G., & Bauer, P. (1999) · Controlled Clinical Trials · doi:10.1016/s0197-2456(99)00029-x · Cited by 23 · see citing works
The pros and cons of funnel plots as an aid to risk communication and patient decision making.
Rakow, T., Wright, R. J., Spiegelhalter, D., & Bull, C. (2015) · British Journal of Psychology · doi:10.1111/bjop.12081 · Cited by 23 · see citing works
Management and Conduct of Randomized Controlled Trials
Knatterud, G. L. (2002) · Epidemiologic Reviews · doi:10.1093/epirev/24.1.12 · Cited by 18 · see citing works
Using the CER Hub to ensure data quality in a multi-institution smoking cessation study
Walker, K. L., Kirillova, O., Gillespie, S. E., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002629 · Cited by 14 · see citing works
Rockenschaub, P., Nguyen, V. G., Aldridge, R. W., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034396 · Cited by 13 · see citing works
Manktelow, B. N., & Seaton, S. E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0045723 · Cited by 12 · see citing works
Pérez‐Benito, F. J., Sáez, C., Conejero, J. A., Tortajada, S., Valdivieso, B., & Garcia-Gomez, J. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220369 · Cited by 11 · see citing works
Self-citationHirsch, O., Donner‐Banzhoff, N., Schulz, M., et al. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15092015 · Cited by 4 · see citing works
Lobo, M. F., Oliveira, M., Oliveira, A. R. S., et al. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-45688-7_10 · Cited by 3 · see citing works
Self-citationSouza, J., Caballero, I., Santos, J. V., et al. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-72651-5_19 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Rocha, H. Adeli, L. Reis, S. Costanzo, I. Orovic, F. Moreira (Eds.), Trends and Innovations in Information Systems and Technologies. WorldCIST 2020. Advances in Intelligent Systems and Computing, vol. 1159, Springer, Cham, 2020, pp. 93-103.
- J.V. Santos, R. Novo, J. Souza, F. Lopes, A. Freitas, Transition from ICD-9-CM to ICD-10-CM/PCS in Portugal: An heterogeneous implementation with potential data implications. Health Information Management Journal [epub ahead of print], 2021.
- E. Broberg, R. Snacken, C. Adlhoch, J. Beauté, M. Galinska, D. Pereyaslov, C. Brown, P. Penttinen, WHO European Region and the European Influenza Surveillance Network. Start of the 2014/15 influenza season in Europe: drifted influenza A(H3N2) viruses circulate as dominant subtype, Euro. Surveill. 20(4): 21023 (2015), https://doi.org/10.2807/1560-7917.es2015.20.4.21023. · DOI
Cited by
Sorted by citations
Badidi, E. (2023) · Future Internet · doi:10.3390/fi15110370 · Cited by 113 · see citing works
- Authors
- Souza, J., Caballero, I., Vasco Santos, J., Lobo, M., Pinto, A., Viana, J., Sáez, C., Lopes, F., & Freitas, A.
- Affiliations
- Universidade do Porto (PT)
- University of Castilla-La Mancha (ES)
- Universitat Politècnica de València (ES)
- Corresponding author
- Souza, Júlio
- ORCID
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2022-12
- Volume
- 136
- Pages
- 104242
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4308925501
- Cited by
- 7
- References
- 44
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 5
- FWCI
- 1.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- RiuNet (Universitat Politècnica de València) (submittedVersion, cc-by-nc-nd)
- Funders
- Universitat Politècnica de València (UPV-SUB.2-1302)
- Fundação para a Ciência e a Tecnologia (POCI-01-0145-FEDER-030766, SBPLY/17/180501/000293, SBPLY/21/180501/000061)
- Ministerio de Ciencia e Innovación (PID2020-112540RB-C42)
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Keywords
- data quality, hospital administrative data, temporal variation, funnel plot, statistical process control, hospitalization data, diagnosis-related groups, risk adjustment, acute myocardial infarction, congestive heart failure
- MeSH terms
- Data Accuracy, Hospitalization, Hospitals, Humans, Myocardial Infarction, Portugal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
