The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Biases introduced by filtering electronic health records for patients with “complete data.”
Open accessAbstract
Objective: One promise of nationwide adoption of electronic health records (EHRs) is the availability of data for large-scale clinical research studies. However, because the same patient could be treated at multiple health care institutions, data from only a single site might not contain the complete medical history for that patient, meaning that critical events could be missing. In this study, we evaluate how simple heuristic checks for data “completeness” affect the number of patients in the resulting cohort and introduce potential biases. Materials and Methods: We began with a set of 16 filters that check for the presence of demographics, laboratory tests, and other types of data, and then systematically applied all 216 possible combinations of these filters to the EHR data for 12 million patients at 7 health care systems and a separate payor claims database of 7 million members. Results: EHR data showed considerable variability in data completeness across sites and high correlation between data types. For example, the fraction of patients with diagnoses increased from 35.0% in all patients to 90.9% in those with at least 1 medication. An unrelated claims dataset independently showed that most filters select members who are older and more likely female and can eliminate large portions of the population whose data are actually complete. Discussion and Conclusion: As investigators design studies, they need to balance their confidence in the completeness of the data with the effects of placing requirements on the data on the resulting patient cohort.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Filtering for "complete" EHRs, a common strategy for managing missing data, introduced a bias toward older patients who were more likely female.
“work by Weber et al. found that simply filtering for “complete” EHRs, a common strategy for managing missing data, introduced a bias toward older patients who were more likely female (50).”
From Accessing Artificial Intelligence for Clinical Decision-Making (2021) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)
Murphy, S. N., Weber, G. M., Mendis, M., Gainer, V. S., Chueh, H. C., Churchill, S. E., & Kohane, I. S. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.000893 · Cited by 926
Self-citationHersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567
Self-citationDefining and measuring completeness of electronic health records for secondary use
Weiskopf, N. G., Hripcsak, G., Swaminathan, S., & Weng, C. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.010 · Cited by 367
Finding the Missing Link for Big Biomedical Data
Weber, G. M., Mandl, K. D., & Kohane, I. S. (2014) · JAMA · doi:10.1001/jama.2014.4228 · Cited by 349
Self-citationSecondary Use of EHR: Data Quality Issues and Informatics Opportunities.
(2010) · PubMed Central · Cited by 332
PCORnet: turning a dream into reality
Collins, F. S., Hudson, K., Briggs, J. P., & Lauer, M. S. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002864 · Cited by 182
Adler‐Milstein, J., Bates, D. W., & Jha, A. K. (2013) · Health Affairs · doi:10.1377/hlthaff.2013.0124 · Cited by 145
Patients Treated at Multiple Acute Health Care Facilities: Quantifying Information Fragmentation
Bourgeois, F. C. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.439 · Cited by 140
Rusanov, A., Weiskopf, N. G., Wang, S., & Weng, C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-51 · Cited by 137
Wei, W.-Q., Leibson, C. L., Ransom, J. E., Kho, A. N., Caraballo, P. J., Chai, H. S., Yawn, B. P., Pacheco, J. A., & Chute, C. G. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000597 · Cited by 112
DeVoe, J. E., Gold, R. B., McIntire, P., Puro, J. E., Chauvie, S., & Gallia, C. A. (2011) · The Annals of Family Medicine · doi:10.1370/afm.1279 · Cited by 104
PCORI at 3 Years — Progress, Lessons, and Plans
Selby, J. V., & Lipstein, S. H. (2014) · New England Journal of Medicine · doi:10.1056/nejmp1313061 · Cited by 103
Lewis, V. A., Colla, C. H., Carluzzo, K. L., Kler, S. E., & Fisher, E. S. (2013) · Health Services Research · doi:10.1111/1475-6773.12102 · Cited by 98
Scalable Collaborative Infrastructure for a Learning Healthcare System (SCILHS): Architecture
Mandl, K. D., Kohane, I. S., McFadden, G., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002727 · Cited by 92
Self-citationHealth information exchange among U.S. hospitals: who's in, who's out, and why?
Adler‐Milstein, J., & Jha, A. K. (2014) · Healthcare · doi:10.1016/j.hjdsi.2013.12.005 · Cited by 80
Mowat, F. S., Lau, Whyte, Kelsh, Legg, Engel-Nitz, Watson, Collins, & Nordyke, R. J. (2011) · Clinical Epidemiology · doi:10.2147/clep.s23690 · Cited by 75
Finnell, J. T., Overhage, J. M., & Grannis, S. J. (2011) · PubMed · Cited by 69
Factors Related to Health Information Exchange Participation and Use
Yeager, V. A., Walker, D. M., Cole, E. S., Mora, A. M., & Diana, M. L. (2014) · Journal of Medical Systems · doi:10.1007/s10916-014-0078-1 · Cited by 63
Emergency Physicians' Perspectives on Their Use of Health Information Exchange
Thorn, S. A., Carter, M. A., & Bailey, J. E. (2013) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2013.09.024 · Cited by 57
Heintzman, J., Bailey, S. R., Hoopes, M. J., Le, T., Gold, R. B., O'Malley, J. P., Cowburn, S., Marino, M., Krist, A. H., & DeVoe, J. E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002333 · Cited by 42
Wei, W.-Q., Leibson, C. L., Ransom, J. E., Kho, A. N., & Chute, C. G. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.05.015 · Cited by 40
Accountable Care Organizations and Antitrust
Shortell, S. M. (2012) · JAMA · doi:10.1001/jama.2012.451 · Cited by 25
Flagged first, then by citations
Cited by
Sorted by citations
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326
Electronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171
Hindorff, L. A., Bonham, V. L., & Ohno-Machado, L. (2018) · Personalized Medicine · doi:10.2217/pme-2018-0037 · Cited by 58
Zhou, Y., Shi, J., Stein, R., Liu, X., Baldassano, R. N., Forrest, C. B., Chen, Y., & Huang, J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad066 · Cited by 33
Metadata
- Authors
- Weber, G. M., Adams, W. G., Bernstam, E. V., Bickel, J. P., Fox, K. P., Marsolo, K., Raghavan, V. A., Turchin, A., Zhou, X., Murphy, S. N., & Mandl, K. D.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Boston Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Boston Children's Hospital (US)
- Aetna (United States) (US)
- Cincinnati Children's Hospital Medical Center (US)
- Merck & Co., Inc., Rahway, NJ, USA (United States) (US)
- Brigham and Women's Hospital (US)
- Wake Forest University Health Sciences (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Weber, Griffin M
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-08-01
- Volume
- 24
- Issue
- 6
- Pages
- 1134-1141
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6080680
- OpenAlex
- W2741416490
- Cited by
- 93
- References
- 22
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 6 · 2023: 13 · 2022: 10 · 2021: 11 · 2020: 15 · 2019: 8 · 2018: 7 · 2017: 1
- FWCI
- 11.65
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute of General Medical Sciences (5R01GM104303)
- National Institutes of Health (U54HG007963, U01CA198934)
- Patient-Centered Outcomes Research Institute (CDRN130604608)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, data completeness, data filtering, selection bias, claims data
- MeSH terms
- Data Accuracy, Humans, Insurance Claim Reporting, Bias, Information Storage and Retrieval, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
