CoChat
Sign Up
Cited by 26Machine Learning in HealthcareElectronic Health Records SystemsArtificial Intelligence in Healthcare

High-throughput phenotyping with temporal sequences

Open access
Estiri, Hossein, Strasser, Zachary H, Murphy, Shawn N·2021·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocaa288PMC PMC7973443

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: High-throughput electronic phenotyping algorithms can accelerate translational research using data from electronic health record (EHR) systems. The temporal information buried in EHRs is often underutilized in developing computational phenotypic definitions. This study aims to develop a high-throughput phenotyping method, leveraging temporal sequential patterns from EHRs. Materials and Methods: We develop a representation mining algorithm to extract 5 classes of representations from EHR diagnosis and medication records: the aggregated vector of the records (aggregated vector representation), the standard sequential patterns (sequential pattern mining), the transitive sequential patterns (transitive sequential pattern mining), and 2 hybrid classes. Using EHR data on 10 phenotypes from the Mass General Brigham Biobank, we train and validate phenotyping algorithms. Results: Phenotyping with temporal sequences resulted in a superior classification performance across all 10 phenotypes compared with the standard representations in electronic phenotyping. The high-throughput algorithm’s classification performance was superior or similar to the performance of previously published electronic phenotyping algorithms. We characterize and evaluate the top transitive sequences of diagnosis records paired with the records of risk factors, symptoms, complications, medications, or vaccinations. Discussion: The proposed high-throughput phenotyping approach enables seamless discovery of sequential record combinations that may be difficult to assume from raw EHR data. Transitive sequences offer more accurate characterization of the phenotype, compared with its individual components, and reflect the actual lived experiences of the patients with that particular disease. Conclusion: Sequential data representations provide a precise mechanism for incorporating raw EHR records into downstream machine learning. Our approach starts with user interpretability and works backward to the technology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa288

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Temporal information in EHRs is often implicit, inconsistently documented, and embedded within unstructured narratives.

    “temporal information in EHRs is often implicit, inconsistently documented, and embedded within unstructured narratives8.”

    From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [8]

    The Introduction says EHR observations are sampled irregularly and that a diagnosis timestamp may not reflect disease onset. The Background says precise timestamps are often unavailable for discrete clinical data. These passages support difficulty recovering temporal information, but they do not say that temporal information is embedded in unstructured narratives; the Background mentions clinical notes only as a source of keywords. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 72

  1. Zhao J, Papapetrou P, Asker L, et al. Learning from heterogeneous temporal data in electronic health records. J Biomed Inform 2017;65:105-19. doi:10.1016/j.jbi.2016.11.006 · DOI
  2. Hripcsak G, Albers DJ, Perotte A. Exploiting time in electronic health record correlations. J Am Med Informatics Assoc Published Online First: 2011. doi:10.1136/amiajnl-2011-000463 · DOI
  3. Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Informatics Assoc Published Online First: 2013. doi:10.1136/amiajnl-2012-001145 · DOI · Paper page
  4. Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: Retrospective observational study. BMJ Published Online First: 2018. doi:10.1136/bmj.k1479 · DOI · Paper page
  5. Banda JM, Seneviratne M, Hernandez-Boussard T, et al. Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models. Annu Rev Biomed Data Sci 2018;1:53-68. doi:10.1146/annurev-biodatasci-080917-013315 · DOI
  6. Ding DY, Simpson C, Pfohl S, et al. The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data. Pac Symp Biocomput 2019;24:18-29.
  7. Agarwal V, Podchiyska T, Banda JM, et al. Learning statistical models of phenotypes using noisy labeled training data. J Am Med Informatics Assoc 2016;23:1166-73. doi:10.1093/jamia/ocw028 · DOI
  8. Yu S, Chakrabortty A, Liao KP, et al. Surrogate-assisted feature extraction for high- throughput phenotyping. J Am Med Inform Assoc 2017;24:e143-9. doi:10.1093/jamia/ocw135 · DOI
  9. Yu S, Liao KP, Shaw SY, et al. Toward high-throughput phenotyping: Unbiased automated feature extraction and selection from knowledge sources. J Am Med Informatics Assoc 2015;22:993-1000. doi:10.1093/jamia/ocv034 12 · DOI
  10. Moskovitch R, Shahar Y. Classification-driven temporal discretization of multivariate time series. Data Min Knowl Discov Published Online First: 2015. doi:10.1007/s10618-014-0380-z · DOI
  11. Batal I, Valizadegan H, Cooper GF, et al. A temporal pattern mining approach for classifying electronic health record data. ACM Trans Intell Syst Technol Published Online First: 2013. doi:10.1145/2508037.2508044 · DOI
  12. Liu Z, Wu L, Hauskrecht M. Modeling Clinical Time Series Using Gaussian Process Sequences. In: Proceedings of the 2013 SIAM International Conference on Data Mining. 2013. doi:10.1137/1.9781611972832.69 · DOI
  13. Madkour M, Benhaddou D, Tao C. Temporal data representation, normalization, extraction, and reasoning: A review from clinical domain. Comput Methods Programs Biomed 2016;128:52- 68. doi:10.1016/j.cmpb.2016.02.007 · DOI
  14. Moskovitch R, Polubriaginof F, Weiss A, et al. Procedure prediction from symbolic Electronic Health Records via time intervals analytics. J Biomed Inform Published Online First: 2017. doi:10.1016/j.jbi.2017.07.018 · DOI
  15. Albers DJ, Hripcsak G. Estimation of time-delayed mutual information and bias for irregularly and sparsely sampled time-series. Chaos, Solitons and Fractals Published Online First: 2012. doi:10.1016/j.chaos.2012.03.003 · DOI
  16. Frey LJ, Lenert L, Lopez-Campos G. EHR Big Data Deep Phenotyping. IMIA Yearb 2014;9:206-11. doi:10.15265/IY-2014-0006 · DOI
  17. Shivade C, Raghavan P, Fosler-Lussier E, et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. J Am Med Informatics Assoc 2014;21:221-30. doi:10.1136/amiajnl-2013-001935 · DOI
  18. Liao KP, Ananthakrishnan AN, Kumar V, et al. Methods to develop an electronic medical record phenotype algorithm to compare the risk of coronary artery disease across 3 chronic disease cohorts. PLoS One Published Online First: 2015. doi:10.1371/journal.pone.0136651 · DOI
  19. Liao KP, Cai T, Savova GK, et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. Br Med J Published Online First: 2015. doi:10.1136/bmj.h1885 · DOI
  20. Roden DM, Pulley JM, Basford MA, et al. Development of a large-scale de-identified DNA biobank to enable personalized medicine. Clin Pharmacol Ther Published Online First: 2008. doi:10.1038/clpt.2008.89 · DOI
  21. Bowton EA, Collier SP, Wang X, et al. Phenotype-Driven Plasma biobanking strategies and Methods. J Pers Med Published Online First: 2015. doi:10.3390/jpm5020140 24 · DOI
  22. McCarty CA, Chisholm RL, Chute CG, et al. The eMERGE Network: A consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Med Genomics 2011;4:13. doi:10.1186/1755-8794-4-13 · DOI
  23. Gottesman O, Kuivaniemi H, Tromp G, et al. The Electronic Medical Records and Genomics (eMERGE) Network: Past, present, and future. Genet. Med. 2013. doi:10.1038/gim.2013.72 · DOI
  24. Pathak J, Wang J, Kashyap S, et al. Mapping clinical phenotype data elements to standardized metadata repositories and controlled terminologies: the eMERGE Network experience. J Am Med Inform Assoc;18:376-86. doi:10.1136/amiajnl-2010-000061 · DOI
  25. Kho AN, Pacheco JA, Peissig PL, et al. Electronic medical records for genetic research: Results of the eMERGE consortium. Sci Transl Med Published Online First: 2011. doi:10.1126/scitranslmed.3001807 · DOI · Paper page
  26. Kirby JC, Speltz P, Rasmussen L V., et al. PheKB: A catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Informatics Assoc Published Online First: 2016. doi:10.1093/jamia/ocv202 29 · DOI
  27. Anderson AE, Kerr WT, Thames A, et al. Electronic health record phenotyping improves detection and screening of type 2 diabetes in the general United States population: A cross- sectional, unselected, retrospective study. J Biomed Inform Published Online First: 2016. doi:10.1016/j.jbi.2015.12.006 · DOI
  28. Yu S, Ma Y, Gronsbell J, et al. Enabling phenotypic big data with PheNorm. J Am Med Informatics Assoc 2018;25:54-60. doi:10.1093/jamia/ocx111 · DOI
  29. Liao KP, Cai T, Gainer V, et al. Electronic medical records for discovery research in rheumatoid arthritis. Arthritis Care Res (Hoboken) Published Online First: 2010. doi:10.1002/acr.20184 · DOI
  30. Ananthakrishnan AN, Cai T, Savova G, et al. Improving Case Definition of Crohn's Disease and Ulcerative Colitis in Electronic Medical Records Using Natural Language Processing: A Novel Informatics Approach. Inflamm Bowel Dis Published Online First: 2013. doi:10.1097/MIB.0b013e31828133fd · DOI
  31. Xia Z, Secor E, Chibnik LB, et al. Modeling disease severity in multiple sclerosis using electronic health records. PLoS One Published Online First: 2013. doi:10.1371/journal.pone.0078927 · DOI
  32. Kumar V, Liao K, Cheng S-C, et al. Natural language processing improves phenotypic accuracy in an electronic medical record cohort of type 2 diabetes and cardiovascular disease. J Am Coll Cardiol Published Online First: 2014. doi:10.1016/S0735-1097(14)61359-0 · DOI
  33. Castro VM, Minnier J, Murphy SN, et al. Validation of electronic health record phenotyping of bipolar disorder cases and controls. Am J Psychiatry Published Online First: 2015. doi:10.1176/appi.ajp.2014.14030423 · DOI
  34. Yu S, Kumamaru KK, George E, et al. Classification of CT pulmonary angiography reports by presence, chronicity, and location of pulmonary embolism with natural language processing. J Biomed Inform Published Online First: 2014. doi:10.1016/j.jbi.2014.08.001 · DOI
  35. Banda JM, Halpern Y, Sontag D, et al. Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network. AMIA Jt Summits Transl Sci proceedings AMIA Jt Summits Transl Sci 2017;2017:48- 57.https://pubmed.ncbi.nlm.nih.gov/28815104
  36. Sun W, Rumshisky A, Uzuner O. Temporal reasoning over clinical text: the state of the art. J Am Med Inform Assoc 2013;20:814-9. doi:10.1136/amiajnl-2013-001760 · DOI
  37. Khoshnevisan F, Ivy J, Capan M, et al. Recent temporal pattern mining for septic shock early prediction. In: Proceedings -2018 IEEE International Conference on Healthcare Informatics, ICHI 2018. 2018. doi:10.1109/ICHI.2018.00033 · DOI · Paper page
  38. Shahar Y, Musen MA. Knowledge-based temporal abstraction in clinical domains. Artif Intell Med Published Online First: 1996. doi:10.1016/0933-3657(95)00036-4 · DOI
  39. Stacey M, McGregor C. Temporal abstraction in intelligent clinical data analysis: A survey. Artif Intell Med Published Online First: 2007. doi:10.1016/j.artmed.2006.08.002 · DOI · Paper page
  40. Agrawal R, Srikant R. Mining sequential patterns. In: Proceedings -International Conference on Data Engineering. 1995. doi:10.1109/icde.1995.380415 · DOI
  41. Mabroukeh NR, Ezeife CI. A taxonomy of sequential pattern mining algorithms. ACM Comput Surv Published Online First: 2010.https://dl.acm.org/citation.cfm?id=1824798
  42. Berlingerio M, Bonchi F, Giannotti F, et al. Mining Clinical Data with a Temporal Dimension: A Case Study. 2007 IEEE Int. Conf. Bioinforma. Biomed. (BIBM 2007). 2007;:429-36. doi:10.1109/BIBM.2007.42 · DOI
  43. Zaki MJ. Parallel Sequence Mining on Shared-Memory Machines. J Parallel Distrib Comput 2001;61:401-26. doi:10.1006/jpdc.2000.1695 · DOI
  44. Ayres J, Flannick J, Gehrke J, et al. Sequential PAttern Mining Using a Bitmap Representation. Proc. Eighth ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. 2002;:429-35. doi:10.1145/775047.775109 · DOI
  45. Shknevsky A, Shahar Y, Moskovitch R. Consistent discovery of frequent interval-based temporal patterns in chronic patients' data. J Biomed Inform 2017;75:83-95. doi:10.1016/j.jbi.2017.10.002 · DOI
  46. Estiri H, Strasser ZH, Klann JG, et al. Transitive Sequencing Medical Records for Mining Predictive and Interpretable Temporal Representations. Patterns Published Online First: 22 June 2020. doi:10.1016/j.patter.2020.100051 · DOI
  47. Meyer PE. Information-theoretic variable selection and network inference from microarray data. Ph D Thesis Univ Libr Bruxelles Published Online First: 2008.https://dipot.ulb.ac.be/dspace/bitstream/2013/210396/1/baf3a39e-3c11-496d-8b3b- b952a1827ca0.txt
  48. Cover TM, Thomas JA. Elements of Information Theory. 2nd ed. Hoboken, New Jersey.: : John Wiley & Sons 2012.
  49. Paninski L. Estimation of Entropy and Mutual Information. Neural Comput 2003;15:1191-253. doi:10.1162/089976603321780272 · DOI · Paper page
  50. Yang HH, Moody J. Feature selection based on joint mutual information. In: Proceedings of International ICSC Symposium on Advances in Intelligent Data Analysis. Rochester, New York: 1999. https://pdfs.semanticscholar.org/dd69/1540c3f28decb477a7738f16aa92709b0f59.pdf
  51. Chiu PH, Hripcsak G. EHR-based phenotyping: Bulk learning and evaluation. J Biomed Inform 2017;70:35-51. doi:10.1016/j.jbi.2017.04.009 · DOI
  52. Zheng T, Xie W, Xu L, et al. A machine learning-based framework to identify type 2 diabetes through electronic health records. Int J Med Inform 2017;97:120-7. doi:10.1016/j.ijmedinf.2016.09.014 · DOI
  53. Kartoun U, Aggarwal R, Beam AL, et al. Development of an Algorithm to Identify Patients with Physician-Documented Insomnia. Sci Rep 2018;8. doi:10.1038/s41598-018-25312-z · DOI
  54. Ning W, Chan S, Beam A, et al. Feature extraction for phenotyping from semantic and knowledge resources. J Biomed Inform 2019;91:103122. doi:https://doi.org/10.1016/j.jbi.2019.103122 · DOI
  55. Liao KP, Sun J, Cai TA, et al. High-throughput Multimodal Automated Phenotyping (MAP) with Application to PheWAS. bioRxiv. 2019. doi:10.1101/587436 · DOI
  56. Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care 2010;48:S106-13. doi:10.1097/MLR.0b013e3181de9e17 · DOI
  57. Liu C, Wang F, Hu J, et al. Temporal Phenotyping from Longitudinal Electronic Health Records. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining -KDD '15. Sydney NSW Australia: : Association for Computing Machinery 2015. 705-714. doi:10.1145/2783258.2783352 · DOI
  58. Gottesman RF, Albert MS, Alonso A, et al. Associations between midlife vascular risk factors and 25-year incident dementia in the Atherosclerosis Risk in Communities (ARIC) cohort. JAMA Neurol 2017;74:1246-54. doi:10.1001/jamaneurol.2017.1658 · DOI · Paper page
  59. Whitmer RA, Sidney S, Selby J, et al. Midlife cardiovascular risk factors and risk of dementia in late life. Neurology 2005;64:277-81. doi:10.1212/01.WNL.0000149519.47454.F2 · DOI · Paper page
  60. Freitag MH, Peila R, Masaki K, et al. Midlife pulse pressure and incidence of dementia: The Honolulu-Asia Aging Study. Stroke 2006;37:33-7. doi:10.1161/01.STR.0000196941.58869.2d · DOI
  61. Smith GL, Lichtman JH, Bracken MB, et al. Renal Impairment and Outcomes in Heart Failure. Systematic Review and Meta-Analysis. J Am Coll Cardiol 2006;47:1987-96. doi:10.1016/j.jacc.2005.11.084 · DOI
  62. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature Published Online First: 1986. doi:10.1038/323533a0 · DOI · Paper page
  63. Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Comput Published Online First: 1997. doi:10.1162/neco.1997.9.8.1735 · DOI · Paper page
  64. Huang S-H, Cheng C-H, Chiang C-H, et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Kyunghyun. Circuits Syst 2006 APCCAS 2006 IEEE Asia Pacific Conf Published Online First: 2006. doi:10.1109/APCCAS.2006.342179 · DOI
  65. Choi E, Schuetz A, Stewart WF, et al. Using recurrent neural network models for early detection of heart failure onset. J Am Med Informatics Assoc Published Online First: 2017. doi:10.1093/jamia/ocw112 · DOI · Paper page
  66. Che Z, Purushotham S, Cho K, et al. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Sci Rep Published Online First: 2018. doi:10.1038/s41598-018-24271-9 · DOI · Paper page
  67. Lipton ZC, Kale DC, Elkan C, et al. Learning to Diagnose with LSTM Recurrent Neural Networks. CoRR 2015;abs/1511.0.http://arxiv.org/abs/1511.03677
  68. Pham T, Tran T, Phung D, et al. Predicting healthcare trajectories from medical records: A deep learning approach. J Biomed Inform Published Online First: 2017. doi:10.1016/j.jbi.2017.04.001 · DOI · Paper page
  69. Jin B, Che C, Liu Z, et al. Predicting the Risk of Heart Failure With EHR Sequential Data Modeling. IEEE Access 2018;6:9256-61. doi:10.1109/ACCESS.2017.2789324 · DOI
  70. Jagannatha AN, Yu H. Bidirectional RNN for Medical Event Detection in Electronic Health Records. Proc Conf Assoc Comput Linguist North Am Chapter Meet 2016;2016:473-82.
  71. Choi E, Bahadori MT, Schuetz A, et al. Doctor AI: Predicting Clinical Events via Recurrent Neural Networks. 2016;:301-18.http://proceedings.mlr.press/v56/Choi16.pdf
  72. Zhang J, Kowsari K, Harrison JH, et al. Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record. IEEE Access. 2018. doi:10.1109/ACCESS.2018.2875677 · DOI

Cited by

Sorted by citations

Authors
Estiri, H., Strasser, Z. H., & Murphy, S. N.
Affiliations
Harvard University (US)
Corresponding author
Estiri, Hossein
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2020-12-14
Volume
28
Issue
4
Pages
772-781
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocaa288
PMC
PMC7973443
OpenAlex
W3113123892
Cited by
26
References
2
Citations by year
2026: 4 · 2025: 1 · 2024: 6 · 2023: 10 · 2022: 3 · 2021: 2
FWCI
1.47
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
MSMR algorithm
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
electronic phenotyping, temporal sequence, sequential pattern mining, electronic health records, phenotype classification, model interpretability, high-throughput phenotyping
MeSH terms
Machine Learning, Algorithms, Diagnosis, Drug Therapy, Humans, Time Factors, Data Mining, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review