Developing Deep Learning Continuous Risk Models for Early Adverse Event Prediction in Electronic Health Records: an AKI Case Study
Open accessAbstract
Early detection of patient deterioration is key to unlocking the potential for targeted preventative care and improving patient outcomes. This protocol describes a workflow for developing deep learning continuous risk models for early prediction of future acute adverse events from electronic health records (EHR), taking the prediction of the risk of future acute kidney injury (AKI) as an exemplar. The protocol consists of 34 steps grouped into the following stages: formal problem definition, data processing, model architecture selection, risk calibration and uncertainty, and evaluating model generalisability. For the protocol to be applicable to modelling the future risk of a particular condition, the problem formulation should be clinically and physiologically plausible and there needs to be sufficient associated predictive signal in routinely collected EHR data. Prospective validation is key in evaluating whether retrospective models developed by following the proposed protocol are clinically applicable and useful.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Continuous risk models identified 55.8% of acute kidney injury cases up to 48 h early with a false positive rate of 2:1.
“Continuous risk models identified 55.8% of AKI cases up to 48 h early with a false positive rate of 2:1.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [71]
The model correctly predicted 90.2% of acute kidney injury cases requiring dialysis within 90 days.
“The model correctly predicted 90.2% of AKI cases requiring dialysis”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [71]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
KDIGO Clinical Practice Guidelines for Acute Kidney Injury
Khwaja, A. (2012) · Nephron Clinical Practice · doi:10.1159/000339789 · Cited by 8,039 · see citing works
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · see citing works
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., et al. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,848 · see citing works
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · see citing works
Self-citationAn Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,281 · see citing works
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · see citing works
Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.
Rhee, C., Dantes, R., Epstein, L., Murphy, D. J., Seymour, C. W., Iwashyna, T. J., Kadri, S. S., Angus, D. C., Danner, R. L., Fiore, A. E., et al. (2017) · JAMA · doi:10.1001/jama.2017.13836 · Cited by 1,884 · see citing works
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710 · see citing works
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · see citing works
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504 · see citing works
Hybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412 · see citing works
Self-citationA clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086 · see citing works
Self-citationGraves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598 · see citing works
Acute Kidney Injury and Mortality in Hospitalized Patients
Wang, H. E., Muntner, P., Chertow, G. M., & Warnock, D. G. (2012) · American Journal of Nephrology · doi:10.1159/000337487 · Cited by 539 · see citing works
Quasi-Recurrent Neural NetworksPreprint
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01576 · Cited by 481 · see citing works
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S. M., Downing, N. L., Ng, A. Y., & Shah, N. H. (2017) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2017.8217669 · Cited by 445 · see citing works
Machine Learning and Decision Support in Critical Care
Johnson, A. E. W., Ghassemi, M. M., Nemati, S., Niehaus, K. E., Clifton, D. A., & Clifford, G. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2501978 · Cited by 438 · see citing works
Risk Prediction with Electronic Health Records: A Deep Learning Approach
Cheng, Y., Wang, F., Zhang, P., & Hu, J. (2016) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974348.49 · Cited by 404 · see citing works
Bihorac, A., Ozrazgat-Baslanti, T., Ebadi, A., Motaei, A., Madkour, M., Pardalos, P. M., Lipori, G., Hogan, W. R., Efron, P. A., Moore, F., et al. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000002706 · Cited by 377 · see citing works
Wilson, F. P., Shashaty, M. G. S., Testani, J. M., Aqeel, I., Borovskiy, Y., Ellenberg, S. S., Feldman, H. I., Fernández, H., Gitelman, Y., Lin, J. J., et al. (2015) · The Lancet · doi:10.1016/s0140-6736(15)60266-5 · Cited by 353 · see citing works
The development of a machine learning inpatient acute kidney injury prediction model.
Koyner, J. L., Carey, K. A., Edelson, D. P., & Churpek, M. M. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003123 · Cited by 323 · see citing works
The economic impact of acute kidney injury in England
Kerr, M., Bedford, M., Matthews, B. B., & O'Donoghue, D. J. (2014) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfu016 · Cited by 273 · see citing works
Mohamadlou, H., Lynn-Palevsky, A., Barton, C., Chettipally, U. K., Shieh, L. Y., Calvert, J. S., Saber, N., & Das, R. (2018) · Canadian Journal of Kidney Health and Disease · doi:10.1177/2054358118776326 · Cited by 161 · see citing works
Relational recurrent neural networksPreprint
Santoro, A., Faulkner, R., Raposo, D. C., Rae, J. W., Chrzanowski, M., Weber, T., Wierstra, D., Vinyals, O., Pascanu, R., & Lillicrap, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01822 · Cited by 153 · see citing works
Self-citationWhy the C-statistic is not informative to evaluate early warning scores and what metrics to use
Brufau, S. R., Huddleston, J. M., Escobar, G. J., & Liebow, M. (2015) · Critical Care · doi:10.1186/s13054-015-0999-1 · Cited by 141 · see citing works
Perotte, A., Ranganath, R., Hirsch, J. S., Blei, D., & Elhadad, N. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv024 · Cited by 128 · see citing works
Lei, T., Zhang, Y., & Artzi, Y. (2017) · Cited by 121 · see citing works
Development of a Multicenter Ward–Based AKI Prediction Model
Koyner, J. L., Adhikari, R., Edelson, D. P., & Churpek, M. M. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.00280116 · Cited by 113 · see citing works
Causal inference from observational data
Listl, S., Jürges, H., & Watt, R. G. (2016) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12231 · Cited by 75 · see citing works
Use of Dummy Variables in Regression Equations
Suits, D. B. (1957) · Journal of the American Statistical Association · doi:10.2307/2281705 · Cited by 71 · see citing works
NCEPOD report on acute kidney injury—must do better
MacLeod, A. M. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61843-2 · Cited by 68 · see citing works
Predicting Inpatient Acute Kidney Injury over Different Time Horizons: How Early and Accurate?
Cheng, P., Waitman, L. R., Hu, Y. Q., & Liu, M. (2017) · PubMed Central · Cited by 51 · see citing works
Steppe, J. M., & Bauer, K. W. (1997) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(97)00059-x · Cited by 46 · see citing works
Soleimani, H., Subbaswamy, A., & Saria, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02038 · Cited by 23 · see citing works
Weisenthal, S. J., Quill, C. M., Farooq, S. A., Kautz, H., & Zand, M. S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0204920 · Cited by 15 · see citing works
Personalized Risk Scoring for Critical Care Patients using Mixtures of Gaussian Process ExpertsPreprint
Alaa, A. M., Yoon, J., Hu, S. B., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.00959 · Cited by 12 · see citing works
A Self-Correcting Deep Learning Approach to Predict Acute Conditions in Critical CarePreprint
Pan, Z., Du, H. Y., Ngiam, K. Y., Wang, F., Shum, P., & Feng, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04364 · Cited by 10 · see citing works
Nezhad, M. Z., Zhu, D., Sadati, N., & Yang, K. (2018) · arXiv (Cornell University) · Cited by 7 · see citing works
Disease-Atlas: Navigating Disease Trajectories with Deep LearningPreprint
Lim, B., & van der Schaar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10254 · Cited by 6 · see citing works
Weisenthal, S. J., Liao, H., Ng, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01880 · Cited by 4 · see citing works
Causal Inference in Observational DataPreprint
Yadav, P., Prunelli, L., Hoff, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04660 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 18
- M. Komorowski, L. A. Celi, O. Badawi, A. Gordon, and A. Faisal, "The arti cial intelligence clinician learns optimal treatment strategies for sepsis in intensive care," Nature Medicine, vol. 24, pp. 1716-1720, 2018.
- J. Futoma, S. Hariharan, and K. A. Heller, "Learning to detect sepsis with a multitask gaussian process RNN classi er," in Proceedings of the International Conference on Machine Learning (D.
- P. Nguyen, T. Tran, N. Wickramasinghe, and S. Venkatesh, "Deepr: A convolutional net for medical records," IEEE Journal of Biomedical and Health Informatics, vol. 21, no. 1, pp. 22-30, 2017.
- R. M. Cronin, J. P. VanHouten, E. D. Siew, S. K. Eden, S. D. Fihn, C. D. Nielson, J. F. Peterson, C. R. Baker, T. A. Ikizler, T. Speroff, and M. E. Matheny, "National Veterans Health Administration inpatient risk strati cation models for hospital acquired acute kidney injury," Journal of the American Medical Informatics Association, vol. 22, no. 5, pp. 1054-1071, 2015.
- M. Singer, C. S. Deutschman, C. W. Seymour, M. Shankar-Hari, D. Annane, M. Bauer, R. Bellomo, G. R. Bernard, J.-D. Chiche, C. M. Coopersmith,et al., "The third international consensus de nitions for sepsis and septic shock (sepsis-3)," Jama, vol. 315, no. 8, pp. 801-810, 2016.
- Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, "E cient backprop," in Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, (London, UK), pp. 9-50, Springer-Verlag, 1998.
- X. Glorot and Y. Bengio, "Understanding the di culty of training deep feed forward neural networks," in International Conference on Arti cial Intelligence and Statistics (Y. W. Teh and M. Titterington, eds.), vol. 9, pp. 249-256, 2010.
- B. Zadrozny and C. Elkan, "Transforming classi er scores into accurate multiclass probability estimates," in Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 694-699, ACM, 2002.
- G. W. Brier, "Veri cation of forecasts expressed in terms of probability," Monthly Weather Review, vol. 78, no. 1, pp. 1-3, 1950.
- E. J. Topol, "High-performance medicine: the convergence of human and arti cial intelligence," Nat. Med., vol. 25, pp. 44-56, Jan 2019.
- J. Collins, J. Sohl-Dickstein, and D. Sussillo, "Capacity and learnability in recurrent neural networks," International Conference on Learning Representations, 2017.
- S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
- X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse recti er neural networks," in Proceedings of the 14th International Conference on Arti cial Intelligence and Statistics (G. Gordon, D. Dunson, and M. Dudík, eds.), vol. 15, pp. 315-323, PMLR, 2011.
- A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Recti er nonlinearities improve neural network acoustic models," in Proceedings of the International Conference on Machine Learning (S. Dasgupta and D. McAllester, eds.), vol. 30, p. 3, 2013.
- P. Ramachandran, B. Zoph, and Q. V. Le, "Searching for activation functions," International Conference on Learning Representations, 2018.
- G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, "Self-normalizing neural networks," in Advances in Neural Information Processing Systems (I.
- Guyon, U. Luxburg,1000S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, pp. 971-980, 2017.
- M. Basirat and P. M. Roth, "The quest for the golden activation function," arXiv Preprint arXiv:1808.00783, 2018. · arXiv
Cited by
Sorted by citations
Badidi, E. (2023) · Future Internet · doi:10.3390/fi15110370 · Cited by 113 · see citing works
- Authors
- Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., Connell, A., Hughes, C. O., Karthikesalingam, A., Cornebise, J., Montgomery, H., Rees, G., Laing, C., Baker, C. R., Peterson, K., Reeves, R., Hassabis, D., King, D., Suleyman, M., Back, T., Nielson, C., Ledsam, J. R., & Mohamed, S.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- University College London (GB)
- University College Hospital (GB)
- United States Department of Veterans Affairs (US)
- VA Salt Lake City Healthcare System (US)
- ORCID
- Venue
- Research Square
- Published
- 2019-07-31
- OpenAlex
- W2964389652
- Cited by
- 3
- References
- 52
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- Research Square (acceptedVersion, cc-by)
- Funders
- U.S. Department of Veterans Affairs
- DeepMind
- National Institute for Health and Care Research
- University College London
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- acute kidney injury, deep learning, electronic health records, clinical deterioration, risk calibration, uncertainty estimation, model generalization, prospective validation, clinical risk prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
