CoChat
Sign Up
Cited by 75Machine Learning in HealthcareArtificial Intelligence in HealthcareDiabetes Management and Research

Predicting complications of diabetes mellitus using advanced machine learning algorithms

Open access
Ljubic, Branimir, Hai, Ameen Abdel, Stanojevic, Marija, Diaz, Wilson, Polimac, Daniel, Pavlovski, Martin, Obradovic, Zoran·2020·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocaa120PMC PMC7647294
Open this paper in CoChat — free account

Abstract

Objective: We sought to predict if patients with type 2 diabetes mellitus (DM2) would develop 10 selected complications. Accurate prediction of complications could help with more targeted measures that would prevent or slow down their development. Materials and Methods: Experiments were conducted on the Healthcare Cost and Utilization Project State Inpatient Databases of California for the period of 2003 to 2011. Recurrent neural network (RNN) long short-term memory (LSTM) and RNN gated recurrent unit (GRU) deep learning methods were designed and compared with random forest and multilayer perceptron traditional models. Prediction accuracy of selected complications were compared on 3 settings corresponding to minimum number of hospitalizations between diabetes diagnosis and the diagnosis of complications. Results: The diagnosis domain was used for experiments. The best results were achieved with RNN GRU model, followed by RNN LSTM model. The prediction accuracy achieved with RNN GRU model was between 73% (myocardial infarction) and 83% (chronic ischemic heart disease), while accuracy of traditional models was between 66% – 76%. Discussion: The number of hospitalizations was an important factor for the prediction accuracy. Experiments with 4 hospitalizations achieved significantly better accuracy than with 2 hospitalizations. To achieve improved accuracy deep learning models required training on at least 1000 patients and accuracy significantly dropped if training datasets contained 500 patients. The prediction accuracy of complications decreases over time period. Considering individual complications, the best accuracy was achieved on depressive disorder and chronic ischemic heart disease. Conclusions: The RNN GRU model was the best choice for electronic medical record type of data, based on the achieved results.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa120

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Accurate prediction helps to target nursing interventions better.

    “Consequently, accurate prediction helps to target nursing interventions better (Ljubic et al., 2020).”

    From Artificial intelligence based prediction models for individuals at risk of multiple diabetic complications: A systematic review of the literature (2022) · cites this paper as Ljubic et al., 2020

    The Discussion (page not available) says accurate prediction can identify high-risk patients for monitoring and intervention, supporting the general point about targeting interventions. It does not specifically discuss nursing interventions, and it frames the benefit as a potential application rather than an established effect. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In Ljubic et al. (2020), RNN models were most accurate in predicting depressive disorder and IHD.

    “RNN models were most accurate in predicting depressive disorder and IHD.”

    From Artificial intelligence based prediction models for individuals at risk of multiple diabetic complications: A systematic review of the literature (2022) · cites this paper as Ljubic et al. (2020)

    The Discussion says RNN models were most accurate when predicting depressive disorder and ICHD. The cited paper specifies chronic ischemic heart disease, whereas the claim says IHD without that qualifier. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • For four visits in Ljubic et al. (2020), the accuracy was 0.746 ± 0.053 for GRU RNN, 0.746 ± 0.053 for bidirectional GRU, 0.719 ± 0.073 for 1-way LSTM, 0.671 ± 0.033 for RF, and 0.668 ± 0.039 for MLP.

    “We present a comparison for four visits only and the values for the GRU RNN model with the accuracy of 0.746 ± 0.053, bidirectional GRU with 0.746 ± 0.053, 1‐way LSTM with 0.719 ± 0.073, RF with 0.671 ± 0.033 and MLP with 0.668 ± 0.039.”

    From Artificial intelligence based prediction models for individuals at risk of multiple diabetic complications: A systematic review of the literature (2022) · cites this paper as Ljubic et al. (2020)

    The Results (Table 2) report exactly these four-visit accuracies for neuropathy: 0.746 ± 0.053 for bidirectional GRU, 0.719 ± 0.073 for 1-way LSTM, 0.671 ± 0.033 for RF, and 0.668 ± 0.039 for MLP. The paper calls the bidirectional GRU a GRU RNN; it does not report a separate GRU RNN model with the same accuracy. The claim presents one model as two. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Ljubic et al. (2020) did not present AUC but gave accuracy for all models at four, three, and two visits.

    “Ljubic et al. (2020) did not present the AUC in their paper but only gave the accuracy for all models at four, three and two visits.”

    From Artificial intelligence based prediction models for individuals at risk of multiple diabetic complications: A systematic review of the literature (2022) · cites this paper as Ljubic et al. (2020)

    Materials and Methods identifies accuracy as the evaluation metric, and the Results report accuracy for all four models at two, three, and four visits. The claim that the paper gave only accuracy is too broad: Results Table 3 also reports sensitivity and specificity for the bidirectional GRU model in the four-visit scenario. The searched passages did not show an AUC result. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Ljubic et al. reported that an RNN using hospitalization longitudinal data to predict development of 10 selected complications in T2D achieved prediction accuracy between 73% (myocardial infarction) and 83% (chronic ischemic heart disease), while traditional models achieved between 66% and 76%.

    “the prediction accuracy was between 73% (myocardial infarction) and 83% (chronic ischemic heart disease), while the accuracy of traditional models was between 66% and 76%”

    From Artificial intelligence in diabetes management: Advancements, opportunities, and challenges (2023) · cites this paper as Ljubic et al.99

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1]1. Orasanu G, Plutzky J. The pathologic continuum of diabetic vascular disease. J Am Coll Cardiol 2009; 53 (5): S35–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. [2]2. Deedwania PC. Management of patients with stable angina and type 2 diabetes. Rev Cardiovasc Med 2015; 16 (2): 105–13. [DOI] [PubMed] [Google Scholar]
  3. [7]7. Rodriguez-Gutierrez R, Gonzalez-Gonzalez JG, Zuñiga-Hernandez JA, et al. Benefits and harms of intensive glycemic control in patients with type 2 diabetes. BMJ 2019; 367: l5887. [DOI] [PubMed] [Google Scholar]
  4. [8]8. Garber AJ, Abrahamson MJ, Barzilay JI, et al. Consensus statement by the American Association of Clinical Endocrinologists and American College of Endocrinology on the comprehensive type 2 Diabetes management Algorithm - 2018 Executive summary. Endocr Pract 2018; 24 (1): 91–120. [DOI] [PubMed] [Google Scholar]
  5. [9]9. Qureshi M, Gammoh E, Shakil J, et al. Update on management of type 2 diabetes for cardiologists. Methodist Debakey Cardiovasc J 2018; 14 (4): 273–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. [10]10. Bailey CJ, Day C. Treatment of type 2 diabetes: future approaches. Br Med Bull 2018; 126 (1): 123–37. [DOI] [PubMed] [Google Scholar]
  7. [11]11. Cahn A, Shoshan A, Sagiv T, et al. Use of a Machine Learning Algorithm Improves Prediction of Progression to Diabetes. Diabetes 2018; 67 (suppl 1): db18-1286-P. [Google Scholar]
  8. [12]12. Contreras I, Vehi J. Artificial intelligence for diabetes management and decision support: literature review. J Med Internet Res 2018; 20 (5): e10775. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. [13]13. Bodman MA, Varacallo M. Diabetic Neuropathy. Treasure Island, FL: StatPearls Publishing; 2020. [PubMed] [Google Scholar]
  10. [14]14. Callaghan BC, Cheng H, Stables CL, et al. Diabetic neuropathy: clinical manifestations and current treatments. Lancet Neurol 2012; 11 (6): 521–34. doi: 10.1016/S1474-4422(12)70065-0. [DOI] [PMC free article] [PubMed] [Google Scholar] · DOI
  11. [15]15. Bouhairie VE, McGill JB. Diabetic kidney disease. Mo Med 2016; 113 (5): 390–4. [PMC free article] [PubMed] [Google Scholar]
  12. [16]16. Holt RIG, de Groot M, Golden SH. Diabetes and depression. Curr Diab Rep 2014; 14 (6): 491. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. [17]17. Konrad-Martin D, Reavis KM, Austin D. Hearing impairment in relation to severity of diabetes in a Veteran cohort. Ear Hear 2015; 36 (4): 381–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. [20]20. Hayeri A. Predicting future glucose fluctuations using machine learning and wearable sensor data. Diabetes 2018; 67 (suppl 1): db18-738-P. [Google Scholar]
  15. [21]21. Massaro A, Maritati V, Giannone D, et al. LSTM DSS automatism and dataset optimization for diabetes prediction. Appl Sci 2019; 9 (17): 3532. [Google Scholar]
  16. [22]22. Perveen S, Shahbaz M, Keshavjee K, et al. Prognostic Modeling and Prevention of Diabetes Using Machine Learning Technique. 10.1038/s41598-019-49563-6. [DOI] [PMC free article] [PubMed] · DOI
  17. [23]23. Apoorva S, Aditya SK, Snigdha P, et al. Prediction of diabetes mellitus type-2 using machine learning In: Smys S, Tavares J, Balas V, Iliyasu A, eds. Computational Vision and Bio-Inspired Computing. ICCVBIC 2019. Advances in Intelligent Systems and Computing, vol 1108 Cham, Switzerland: Springer; 2020: 364–70. 10.1007/978-3-030-37218-7_42. [DOI] [Google Scholar] · DOI
  18. [24]24. Sarwar MA, Kamal N, Hamid W, et al. Prediction of diabetes using machine learning Algorithms in healthcare. In: 2018 24th International Conference on Automation and Computing (ICAC); 2018.
  19. [25]25. Zhang XS, Tang F, Dodge H, et al. MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records. In: proceedings of the 25th ACM SIG KDD International Conference on Knowledge Discovery and Data Mining (KDD); 2019: 2487–95. doi.org/10.1145/3292500.3330779. [DOI] [PMC free article] [PubMed] [Google Scholar] · DOI
  20. [26]26. Wang F, Casalino LP, Khullar D. Deep learning in medicine—promise, progress, and challenges. JAMA Intern Med 2019; 179 (3): 293–4. [DOI] [PubMed] [Google Scholar]
  21. [28]28. Klema V, Laub A. The singular value decomposition: its computation and some applications. IEEE Trans Automat Contr 1980; 25 (2): 164–76. [Google Scholar]
  22. [29]29. Williams RJ, Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Comput 1989; 1 (2): 270–80. [Google Scholar]
  23. [30]30. Cho K, van Merrienboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014); 2014; 1724–34.
  24. [31]31. Breiman L. Random forests. Machine Learning 2001; 45 (1): 5–32. doi : 10.1023/A : 1010933404324. [Google Scholar] · DOI
  25. [32]32. Rumelhart DE, Geoffrey EH, Williams RJ. Learning internal representations by error propagation In: Rumelhart DE, McClelland JL, PDP Research Group, eds. Parallel Distributed Processing, Volume1: Explorations in the Microstructure of Cognition: Foundations. Cambridge, MA: MIT Press; 1986: 318–62. [Google Scholar]
  26. [33]33. Ngufor C, Houten HV, Caffo BS, et al. Mixed Effect Machine Learning: a framework for predicting longitudinal change in hemoglobin A1c. J Biomed Inform 2019; 89: 56–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. [34]34. Choi E, Schuetz A, Stewart WF, et al. Using recurrent neural network models for early detection of heart failure onset. J Am Med Inform Assoc 2017; 24 (2): 361–70. [DOI] [PMC free article] [PubMed] [Google Scholar] Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

Cited by

Sorted by citations

Authors
Ljubic, B., Hai, A. A., Stanojevic, M., Diaz, W., Polimac, D., Pavlovski, M., & Obradovic, Z.
Affiliations
  • Temple University (US)
  • Center for Data Analytics and Biomedical Informatics, Temple University, Philadelphia, Pennsylvania, USA
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2020-09-01
Volume
27
Issue
9
Pages
1343-1351
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocaa120
PMC
PMC7647294
OpenAlex
W3082762715
Cited by
75
References
4
Citations by year
2026: 6 · 2025: 15 · 2024: 16 · 2023: 13 · 2022: 16 · 2021: 8 · 2020: 1
FWCI
3.73
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion)
Funders
King Abdullah University of Science and Technology Center Partnership Fund
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Endocrinology, Diabetes and Metabolism › Diabetes Management and Research
Keywords
type 2 diabetes, diabetes complications, complication prediction, recurrent neural networks, gated recurrent unit, long short-term memory, random forest, multilayer perceptron, hospitalization history, myocardial infarction, stable coronary artery disease, depressive disorders, electronic health records, inpatient data
MeSH terms
Deep Learning, Algorithms, Decision Trees, Diabetes Mellitus, Type 2, Humans, Prognosis, Neural Networks, Computer, Risk Assessment, Diabetes Complications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review