Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Datasets
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduction: Data quality and fitness for analysis are crucial if outputs of analyses of electronic health record data or administrative claims data should be trusted by the public and the research community.Methods: We describe a data quality analysis tool (called Achilles Heel) developed by the Observational Health Data Sciences and Informatics Collaborative (OHDSI) and compare outputs from this tool as it was applied to 24 large healthcare datasets across seven different organizations.Results: We highlight 12 data quality rules that identified issues in at least 10 of the 24 datasets and provide a full set of 71 rules identified in at least one dataset. Achilles Heel is a freely available software that provides a useful starter set of data quality rules with the ability to add additional rules. We also present results of a structured email-based interview of all participating sites that collected qualitative comments about the value of Achilles Heel for data quality evaluation.Discussion: Our analysis represents the first comparison of outputs from a data quality tool that implements a fixed (but extensible) set of data quality rules. Thanks to a common data model, we were able to compare quickly multiple datasets originating from several countries in America, Europe and Asia.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542
Electronic Medical Records for Genetic Research: Results of the eMERGE Consortium
Kho, A. N., Pacheco, J. A., Peissig, P. L., Rasmussen, L., Newton, K. M., Weston, N., Crane, P. K., Pathak, J., Chute, C. G., Bielinski, S. J., et al. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3001807 · Cited by 336 · cited 1×
“, missing coding or loss of data),(4) and it can be difficult or impossible to distinguish ETL errors from source data errors.(5,6) In recent years, the biomedical informatics community has increasingly adopted common data models (CDMs) shared across many organizations,(7) because they allow the same analytical code t…”
BackgroundKahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012) · Medical Care · doi:10.1097/mlr.0b013e318257dd67 · Cited by 260
Data Quality Assessment for Comparative Effectiveness Research in Distributed Data Networks
Brown, J. S., Kahn, M., & Toh, S. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1e2c · Cited by 131
Cited by
Electronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171
A Data Quality Assessment Guideline for Electronic Health Record Data Reuse
Weiskopf, N. G., Bakken, S., Hripcsak, G., & Weng, C. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.218 · Cited by 152
Glicksberg, B. S., Oskotsky, B., Thangaraj, P. M., Giangreco, N., Badgeley, M. A., Johnson, K. W., Datta, D., Rudrapatna, V. A., Rappoport, N., Shervey, M. M., et al. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz409 · Cited by 45
Ji, H., Kim, S., Yi, S., Hwang, H., Kim, J.-W., & Yoo, S. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103459 · Cited by 29
Metadata
- Authors
- Huser, V., DeFalco, F. J., Schuemie, M., Ryan, P. B., Shang, N., Velez, M., Park, R. W., Boyce, R. D., Duke, J., Khare, R., Utidjian, L., & Bailey, C.
- Venue
- eGEMs (Generating Evidence & Methods to improve patient outcomes)
- Publisher
- Ubiquity Press, Ltd.
- Published
- 2016-11-30
- Volume
- 4
- Issue
- 1
- Pages
- 24
- ISSN
- 2327-9214 (ISSN-L)
- OpenAlex
- W2557559657
- Cited by
- 61
- References
- 5
- FWCI
- 7.05
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Datasets. https://app.cochat.ai/papers/10.13063/2327-9214.1239
- The paper
- Huser, V., DeFalco, F. J., Schuemie, M., Ryan, P. B., Shang, N., Velez, M., Park, R. W., Boyce, R. D., Duke, J., Khare, R., Utidjian, L., & Bailey, C. (2016). Multisite Evaluation of a Data Quality Tool for Patient-Level Clinical Datasets. eGEMs (Generating Evidence & Methods to improve patient outcomes). https://doi.org/10.13063/2327-9214.1239
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.