CoChat
Sign Up
Cited by 15Machine Learning in HealthcareMedical Coding and Health InformationArtificial Intelligence in Healthcare and Education

Pretrained transformer framework on pediatric claims data for population specific tasks

Open access
Zeng, Xianlong, Linwood, Simon L., Liu, Chang·2022·Scientific Reports
DOI 10.1038/s41598-022-07545-1PMC PMC8901645
Open this paper in CoChat — free accountOpen PDF

Abstract

The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset, followed by a discriminative fine-tuning on each population-specific task. The semantic meaning of medical events can be captured in the pre-training stage, and the effective knowledge transfer is completed through the task-aware fine-tuning stage. The fine-tuning process requires minimal parameter modification without changing the model architecture, which mitigates the data scarcity issue and helps train the deep learning model adequately on small patient cohorts. We conducted experiments on a real-world pediatric dataset with more than one million patient records. Experimental results on two downstream tasks demonstrated the effectiveness of our method: our general task-agnostic pre-training framework outperformed tailored task-specific models, achieving more than 10% higher in model performance as compared to baselines. In addition, our framework showed a potential to transfer learned knowledge from one institution to another, which may pave the way for future healthcare model pre-training across institutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-022-07545-1

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [1]Choi E Schuetz A Stewart WF Sun J Using recurrent neural network models for early detection of heart failure onset J. Am. Med. Inform. Assoc. 2017 24 361 370 10.1093/jamia/ocw112 27521897 PMC5391725 · DOI · Paper page
  2. [2]Landi I Deep representation learning of electronic health records to unlock patient stratification at scale NPJ Digit. Med. 2020 3 1 11 10.1038/s41746-020-0301-z 32699826 PMC7367859 · DOI · Paper page
  3. [3]Zeng, X. et al. Multilevel self-attention model and its use on medical risk prediction. In Pacific Symposium on Biocomputing 2020, 115–126 (World Scientific, 2019). 31797591
  4. [4]Sun, C., Shrivastava, A., Singh, S. & Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE International Conference on Computer Vision, 843–852 (2017).
  5. [5]Hedderich, M. A. & Klakow, D. Training a neural network in a low-resource setting on automatically annotated noisy data. arXiv:1807.00745 (arXiv preprint) (2018). · arXiv
  6. [7]Choi, E. et al. Generating multi-label discrete patient records using generative adversarial networks. In Machine Learning for Healthcare Conference, 286–305 (PMLR, 2017).
  7. [8]Helgheim BI Maia R Ferreira JC Martins AL Merging data diversity of clinical medical records to improve effectiveness Int. J. Environ. Res. Public Health 2019 16 769 10.3390/ijerph16050769 PMC6427263 30832447 · DOI
  8. [9]Seneviratne, M. G., Kahn, M. G. & Hernandez-Boussard, T. Merging heterogeneous clinical data to enable knowledge discovery. In Biocomputing 2019: Proceedings of the Pacific Symposium, 439–443 (World Scientific, 2018). PMC6447393 30864344
  9. [10]Lee D Generating sequential electronic health records using dual adversarial autoencoder J. Am. Med. Inform. Assoc. 2020 27 1411 1419 10.1093/jamia/ocaa119 32989459 PMC7647348 · DOI · Paper page
  10. [11]Buczak AL Babin S Moniz L Data-driven approach for creating synthetic electronic medical records BMC Med. Inform. Decis. Mak. 2010 10 1 28 10.1186/1472-6947-10-59 20946670 PMC2972239 · DOI
  11. [12]Ma, F. et al. Risk prediction on electronic health records with prior medical knowledge. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1910–1919 (2018).
  12. [13]Su, K.-Y., Su, J., Wiebe, J. & Li, H. Proceedings of the joint conference of the 47th annual meeting of the ACL and the 4th international joint conference on natural language processing of the afnlp. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (2009).
  13. [14]Peters, M. E. et al. Deep contextualized word representations. arXiv:1802.05365 (arXiv preprint) (2018). · arXiv
  14. [15]Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L. & Auli, M. Cloze-driven pretraining of self-attention networks. arXiv:1903.07785 (arXiv preprint) (2019). · arXiv
  15. [16]Peters, M. E., Ammar, W., Bhagavatula, C. & Power, R. Semi-supervised sequence tagging with bidirectional language models. arXiv:1705.00108 (arXiv preprint) (2017). · arXiv
  16. [17]Guo, Y. et al. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4805–4814 (2019).
  17. [18]Kan M Wu J Shan S Chen X Domain adaptation for face recognition: Targetize source domain bridged by common subspace Int. J. Comput. Vision 2014 109 94 109 10.1007/s11263-013-0693-1 · DOI
  18. [21]Lee J Biobert: A pre-trained biomedical language representation model for biomedical text mining Bioinformatics 2020 36 1234 1240 31501885 PMC7703786
  19. [22]Alsentzer, E. et al. Publicly available clinical bert embeddings. arXiv:1904.03323 (arXiv preprint) (2019). · arXiv
  20. [23]Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (arXiv preprint) (2018). · arXiv
  21. [24]Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space. arXiv:1301.3781 (arXiv preprint) (2013). · arXiv
  22. [25]He, K., Girshick, R. & Dollár, P. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4918–4927 (2019).
  23. [26]Rasmy L Xiang Y Xie Z Tao C Zhi D Med-bert: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction NPJ Digit. Med. 2021 4 1 13 10.1038/s41746-021-00455-y 34017034 PMC8137882 · DOI · Paper page
  24. [27]Choi, E. et al. Multi-layer representation learning for medical concepts. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1495–1504 (2016).
  25. [28]Su C Machine learning for suicide risk prediction in children and adolescents with electronic health records Transl. Psychiatry 2020 10 1 10 10.1038/s41398-019-0665-5 33243979 PMC7693189 · DOI
  26. [29]Xiang Y Asthma exacerbation prediction and interpretation based on time-sensitive attentive neural network: A retrospective cohort study medRxiv 2019 20 19012161 PMC7428917 32735224
  27. [30]Zeng X Lin S Liu C Multi-view deep learning framework for predicting patient expenditure in healthcare IEEE Open J. Comput. Soc. 2021 2 62 71 10.1109/OJCS.2021.3052518 · DOI
  28. [31]Choi, E. et al. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. arXiv:1608.05745 (arXiv preprint) (2016). · arXiv
  29. [34]Weisberg S Applied Linear Regression 2005 Wiley
  30. [35]Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J. Doctor AI: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, 301–318 (PMLR, 2016). PMC5341604 28286600
  31. [36]Ma, F. et al. Dipole: Diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1903–1911 (2017).
  32. [37]Yüksel, A. E., Türkmen, Y. A., Özgür, A. & Altınel, B. Turkish tweet classification with transformer encoder. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), 1380–1387 (2019).
  33. [38]Nurmagambetov T Kuwahara R Garbe P The economic burden of asthma in the united states, 2008–2013 Ann. Am. Thorac. Soc. 2018 15 348 356 10.1513/AnnalsATS.201703-259OC 29323930 · DOI

Cited by

Sorted by citations

Authors
Zeng, X., Linwood, S. L., & Liu, C.
Affiliations
  • Ohio University (US)
  • Nationwide Children's Hospital (US)
Corresponding author
Liu, Chang
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2022-03-07
Volume
12
Issue
1
Pages
3651-3651
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-022-07545-1
PMC
PMC8901645
OpenAlex
W4220660160
Cited by
15
References
5
Citations by year
2026: 1 · 2025: 6 · 2024: 5 · 2023: 3
FWCI
1.15
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
pre-trained transformer models, electronic health records, fine-tuning, data scarcity, knowledge transfer
MeSH terms
Child, Forecasting, Humans, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review