Temporal relationship of computed and structured diagnoses in electronic health record data
Open accessPublished version of a preprint
Abstract
Background: The electronic health record (EHR) holds the prospect of providing more complete and timely access to clinical information for biomedical research, quality assessments, and quality improvement compared to other data sources, such as administrative claims. In this study, we sought to assess the completeness and timeliness of structured diagnoses in the EHR compared to computed diagnoses for hypertension (HTN), hyperlipidemia (HLD), and diabetes mellitus (DM). Methods: We determined the amount of time for a structured diagnosis to be recorded in the EHR from when an equivalent diagnosis could be computed from other structured data elements, such as vital signs and laboratory results. We used EHR data for encounters from January 1, 2012 through February 10, 2019 from an academic health system. Diagnoses for HTN, HLD, and DM were computed for patients with at least two observations above threshold separated by at least 30 days, where the thresholds were outpatient blood pressure of ≥ 140/90 mmHg, any low-density lipoprotein ≥ 130 mg/dl, or any hemoglobin A1c ≥ 6.5%, respectively. The primary measure was the length of time between the computed diagnosis and the time at which a structured diagnosis could be identified within the EHR history or problem list. Results: We found that 39.8% of those with HTN, 21.6% with HLD, and 5.2% with DM did not receive a corresponding structured diagnosis recorded in the EHR. For those who received a structured diagnosis, a mean of 389, 198, and 166 days elapsed before the patient had the corresponding diagnosis of HTN, HLD, or DM, respectively, recorded in the EHR. Conclusions: We found a marked temporal delay between when a diagnosis can be computed or inferred and when an equivalent structured diagnosis is recorded within the EHR. These findings demonstrate the continued need for additional study of the EHR to avoid bias when using observational data and reinforce the need for computational approaches to identify clinical phenotypes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Documentation timestamps reflect when events are recorded rather than when they occur, introducing systematic errors that can distort event sequencing, bias time-to-event analyses, and undermine model validity.
“Documentation timestamps are often used as surrogates for disease onset; however, they reflect when events are recorded rather than when they occur, introducing systematic errors that can distort event sequencing, bias time-to-event analyses, and undermine model validity9–12.”
From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [11]
The Results (Temporality of clinical and computed diagnoses; page unavailable) show that structured diagnoses were recorded well after diagnoses could be computed from other EHR data. The Discussion (page unavailable) says these delays may bias cohort development, outcome ascertainment, and historical comparisons. The paper does not establish that documentation timestamps generally distort event sequencing, bias time-to-event analyses, or undermine model validity. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · see citing works
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · see citing works
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · see citing works
Flagged first, then by citations
Research works printed but not linked: 37
- [1]Mulder R Singh AB Hamilton A The limitations of using randomised controlled trials as a basis for developing treatment guidelines Evid Based Ment Health 2018 21 1 4 6 10.1136/eb-2017-102701 28710065 PMC10270454 · DOI
- [2]Booth CM Tannock IF Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence Br J Cancer 2014 110 3 551 555 10.1038/bjc.2013.725 24495873 PMC3915111 · DOI
- [3]Sherman RE Anderson SA Dal Pan GJ Real-world evidence—What is it and what can it tell us? N Engl J Med 2016 375 23 2293 2297 10.1056/NEJMsb1609216 27959688 · DOI
- [4]Miksad RA Abernethy AP Harnessing the power of real-world evidence (RWE): a checklist to ensure regulatory-grade data quality Clin Pharmacol Ther 2018 103 2 202 205 10.1002/cpt.946 29214638 PMC5814721 · DOI
- [5]Khosla S, White R, Medina J, et al. Real world evidence (RWE)—a disruptive innovation or the quiet evolution of medical evidence generation? [version 2; peer review: 2 approved]. F1000Res. 2018;7:111. 10.12688/f1000research.13585.2. PMC6039945 30026923 · DOI
- [7]Kim H-S Kim JH Proceed with caution when using real world data and real world evidence J Korean Med Sci 2019 34 4 e28 10.3346/jkms.2019.34.e28 30686950 PMC6345637 · DOI
- [8]Ryan PB Madigan D Stang PE Overhage JM Racoosin JA Hartzema AG Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership Stat Med 2012 31 30 4401 4415 10.1002/sim.5620 23015364 · DOI
- [9]Häyrinen K Saranto K Nykänen P Definition, structure, content, use and impacts of electronic health records: a review of the research literature Int J Med Inform 2008 77 5 291 304 10.1016/j.ijmedinf.2007.09.001 17951106 · DOI
- [11]Mitchell JB, Bubolz T, Paul JE, et al. Using Medicare claims for outcomes research. Med. Care 1994;32(7 Suppl):JS38–51. 8028412
- [12]Blumenthal S. The use of clinical registries in the united states: A landscape survey. EGEMS (Wash. DC) 2017;5(1):26. 10.5334/egems.248. PMC5994955 29930965 · DOI
- [13]Birnbaum HG Cremieux PY Greenberg PE LeLorier J Ostrander JA Venditti L Using healthcare claims data for outcomes research and pharmacoeconomic analyses Pharmacoeconomics 1999 16 1 1 8 10.2165/00019053-199916010-00001 10539118 · DOI
- [14]Hoque DME Kumari V Hoque M Ruseckaite R Romero L Evans SM Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review PLoS ONE 2017 12 9 e0183667 10.1371/journal.pone.0183667 28886607 PMC5591016 · DOI
- [15]Mues KE Liede A Liu J Use of the Medicare database in epidemiologic and health services research: a valuable source of real-world evidence on the older and disabled populations in the US Clin Epidemiol 2017 9 267 277 10.2147/CLEP.S105613 28533698 PMC5433516 · DOI
- [16]Krumholz HM Lin Z Drye EE An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction Circ Cardiovasc Qual Outcomes 2011 4 2 243 252 10.1161/CIRCOUTCOMES.110.957498 21406673 PMC3350811 · DOI
- [17]Ferver K Burton B Jesilow P The use of claims data in healthcare research Open Public Health J 2009 2 1 11 24 10.2174/1874944500902010011 · DOI
- [19]Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann. Intern. Med. 1993;119(8):844–850. 10.7326/0003-4819-119-8-199310150-00011. 8018127 · DOI
- [20]Hartzema AG Racoosin JA MaCurdy TE Gibbs JM Kelman JA Utilizing Medicare claims data for real-time drug safety evaluations:is it feasible? Pharmacoepidemiol Drug Saf 2011 20 7 684 688 10.1002/pds.2143 21847800 · DOI
- [21]Subramanian S Tangka FKL Beebe MC Trebino D Weir HK Babcock F The cost of cancer registry operations: Impact of volume on cost per case for core and enhanced registry activities Eval Program Plann 2016 55 1 8 10.1016/j.evalprogplan.2015.11.005 26702880 PMC4769894 · DOI
- [22]Wright A, McCoy AB, Hickman T-TT, et al. Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int. J. Med. Inform. 2015;84(10):784–790. 10.1016/j.ijmedinf.2015.06.011. PMC4549158 26228650 · DOI
- [23]Singer A Kroeker AL Yakubovich S Duarte R Dufault B Katz A Data quality in electronic medical records in Manitoba: Do problem lists reflect chronic disease as defined by prescriptions? Can Fam Physician 2017 63 5 382 389 28500199 PMC5429058
- [24]Holmes C Brown M Hilaire DS Wright A Healthcare provider attitudes towards the problem list in an electronic health record: a mixed-methods qualitative study BMC Med Inform Decis Mak 2012 12 127 10.1186/1472-6947-12-127 23140312 PMC3534408 · DOI
- [25]Szeto HC Coleman RK Gholami P Hoffman BB Goldstein MK Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic Am J Manag Care 2002 8 1 37 43 11814171
- [26]Smith SM McAuliffe K Hall JM Hypertension in Florida: data from the Oneflorida clinical data research network Prev Chronic Dis 2018 15 E27 10.5888/pcd15.170332 29494332 PMC5833311 · DOI
- [27]Banerjee D Chung S Wong EC Wang EJ Stafford RS Palaniappan LP Underdiagnosis of hypertension using electronic health records Am J Hypertens 2012 25 1 97 102 10.1038/ajh.2011.179 22031453 PMC3600431 · DOI
- [28]Hripcsak G Ryan PB Duke JD Characterizing treatment pathways at scale using the OHDSI network Proc Natl Acad Sci USA 2016 113 27 7329 7336 10.1073/pnas.1510502113 27274072 PMC4941483 · DOI · Paper page
- [29]Toh S Rasmussen-Torvik LJ Harmata EE The National Patient-Centered Clinical Research Network (PCORnet) bariatric study cohort: rationale, methods, and baseline characteristics JMIR Res Protoc 2017 6 12 e222 10.2196/resprot.8323 29208590 PMC5736875 · DOI
- [30]Hripcsak G Albers DJ Next-generation phenotyping of electronic health records J Am Med Inform Assoc 2013 20 1 117 121 10.1136/amiajnl-2012-001145 22955496 PMC3555337 · DOI · Paper page
- [31]Perotte A Hripcsak G Temporal properties of diagnosis code time series in aggregate IEEE J Biomed Health Inform 2013 17 2 477 483 10.1109/JBHI.2013.2244610 24235118 PMC4030411 · DOI
- [32]Elliott AF Davidson A Lum F Use of electronic health records and administrative data for public health surveillance of eye health and vision-related conditions in the United States Am J Ophthalmol 2012 154 6 Suppl S63 70 10.1016/j.ajo.2011.10.002 23158225 PMC4037529 · DOI
- [34]Agniel D Kohane IS Weber GM Biases in electronic health record data due to processes within the healthcare system: retrospective observational study BMJ 2018 361 k1479 10.1136/bmj.k1479 29712648 PMC5925441 · DOI · Paper page
- [35]McPadden J Durant TJ Bunch DR Health care and precision medicine research: analysis of a scalable data science platform J Med Internet Res 2019 21 4 e13043 10.2196/13043 30964441 PMC6477571 · DOI
- [37]Blecker S Katz SD Horwitz LI Comparison of approaches for heart failure case identification from electronic health record data JAMA Cardiol 2016 1 9 1014 1020 10.1001/jamacardio.2016.3236 27706470 PMC5289894 · DOI
- [38]Hripcsak G, Shang N, Peissig PL, et al. Facilitating phenotype transfer using a common data model. J. Biomed. Inform. 2019:103253. 10.1016/j.jbi.2019.103253. PMC6697565 31325501 · DOI
- [40]Weiner MG Embi PJ Toward reuse of clinical data for research and quality improvement: the end of the beginning? Ann Intern Med 2009 151 5 359 360 10.7326/0003-4819-151-5-200909010-00141 19638404 · DOI · Paper page
- [41]Hripcsak G Albers DJ Perotte A Exploiting time in electronic health record correlations J Am Med Inform Assoc 2011 18 Suppl 1 i109 i115 10.1136/amiajnl-2011-000463 22116643 PMC3241180 · DOI
- [42]Golladay KK, Collins AB, Ashcraft A, et al. Adverse Events in Hospitals: Methods for Identifying Events. Department of Health and Human Services; 2010:60.
- [43]Krishnamoorthy A, Peterson ED, Knight JD, et al. How Reliable are Patient-Reported Rehospitalizations? Implications for the Design of Future Practical Clinical Studies. J. Am. Heart Assoc. 2016;5(1). 10.1161/JAHA.115.002695. PMC4859389 26811163 · DOI
Cited by
Sorted by citations
TimeX: Phenotype Onset Extraction from Clinical Narratives
Chen, F., Jiang, S., Nguyen, Q. M., Ta, C. N., Wang, K., Chung, W. K., & Weng, C. (2026) · npj Health Systems · doi:10.1038/s44401-026-00099-8
- Authors
- Schulz, W. L., Young, H. P., Coppi, A., Mortazavi, B. J., Lin, Z., Jean, R. A., & Krumholz, H. M.
- Affiliations
- Yale New Haven Hospital (US)
- Corresponding author
- Krumholz, Harlan M.
- ORCID
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2021-02-17
- Volume
- 21
- Issue
- 1
- Pages
- 61-61
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC7890604
- OpenAlex
- W3133066484
- Cited by
- 20
- References
- 3
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 3 · 2023: 3 · 2022: 3 · 2021: 3 · 2020: 2
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- Interpace Diagnostics
- Shenzhen Center for Health Information
- National Institutes of Health (UL1TR001863)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, time delay, hypertension, hyperlipidemia, diabetes mellitus, clinical phenotype, problem list, data completeness, data timeliness
- MeSH terms
- Diabetes Mellitus, Humans, Hypertension, Outpatients, Information Storage and Retrieval, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
