The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
A clinically applicable approach to continuous prediction of future acute kidney injury
Open accessAbstract
The early prediction of deterioration could have an important role in supporting healthcare professionals, as an estimated 11% of deaths in hospital follow a failure to promptly recognize and treat deteriorating patients 1. To achieve this goal requires predictions of patient risk that are continuously updated and accurate, and delivered at an individual level with sufficient context and enough time to act. Here we develop a deep learning approach for the continuous risk prediction of future deterioration in patients, building on recent work that models adverse events from electronic health records 2-17 and using acute kidney injury-a common and potentially life-threatening condition 18-as an exemplar. Our model was developed on a large, longitudinal dataset of electronic health records that cover diverse clinical environments, comprising 703,782 adult patients across 172 inpatient and 1,062 outpatient sites. Our model predicts 55.8% of all inpatient episodes of acute kidney injury, and 90.2% of all acute kidney injuries that required subsequent administration of dialysis, with a lead time of up to 48 h and a ratio of 2 false alerts for every true alert. In addition to predicting future acute kidney injury, our model provides confidence assessments and a list of the clinical features that are most salient to each prediction, alongside predicted future trajectories for clinically relevant blood tests 9. Although the recognition and prompt treatment of acute kidney injury is known to be challenging, our approach may offer opportunities for identifying patients at risk within a time window that enables early treatment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Many research groups have tackled early prediction of AKI using electronic health records (EHR) data [8–11], but no model so far explains the rationale behind specific predictions despite a clear need for explainable and actionable predictions [6, 12].”
From Machine learning model for early prediction of acute kidney injury (AKI) in pediatric critical care (2021) · cites this paper as [9]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“it was predominantly made up of data from older non-Black men; the average age was 62 years old and over 90% were males.”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [27]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An example of this includes an algorithm developed to predict acute kidney injury (AKI) using clinical data from the US Department of Veteran Affairs [27].”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Though the dataset used to develop the algorithm was large and diverse, containing data from over 700,000 patients from multiple centers”
From Bias in artificial intelligence algorithms and recommendations for mitigation (2023) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Models that use EHR, laboratory, and physiological data predict outcomes such as acute kidney injury [14] and sepsis or critical-care deterioration [15,16].”
In the citing paper
“Model achieves a higher ROC AUC in shorter time windows.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [80]
In the citing paper
“The model achieves a lower PR AUC in shorter time windows.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [80]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
(2010) · Cited by 14,134
Self-citationEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
KDIGO Clinical Practice Guidelines for Acute Kidney Injury
Khwaja, A. (2012) · Nephron Clinical Practice · doi:10.1159/000339789 · Cited by 8,297
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Self-citationScalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504
Hybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412
Self-citationKomorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598
Acute Kidney Injury and Mortality in Hospitalized Patients
Wang, H. E., Muntner, P., Chertow, G. M., & Warnock, D. G. (2012) · American Journal of Nephrology · doi:10.1159/000337487 · Cited by 539
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S. M., Downing, N. L., Ng, A. Y., & Shah, N. H. (2017) · BMC Medical Informatics and Decision Making · doi:10.1109/bibm.2017.8217669 · Cited by 472
Machine Learning and Decision Support in Critical Care
Johnson, A. E. W., Ghassemi, M. M., Nemati, S., Niehaus, K. E., Clifton, D. A., & Clifford, G. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2501978 · Cited by 459
Quasi-Recurrent Neural NetworksPreprint
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01576 · Cited by 423
Risk Prediction with Electronic Health Records: A Deep Learning Approach
Cheng, Y., Wang, F., Zhang, P., & Hu, J. (2016) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611974348.49 · Cited by 403
Bihorac, A., Ozrazgat-Baslanti, T., Ebadi, A., Motaei, A., Madkour, M., Pardalos, P. M., Lipori, G., Hogan, W. R., Efron, P. A., Moore, F., et al. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000002706 · Cited by 377
The development of a machine learning inpatient acute kidney injury prediction model.
Koyner, J. L., Carey, K. A., Edelson, D. P., & Churpek, M. M. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003123 · Cited by 323
Recurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 314
Alge, J. L., & Arthur, J. M. (2014) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.12191213 · Cited by 298
Stenhouse, C. W., Coates, S., Tivey, M., Allsop, P., & Parker, T. (2000) · British Journal of Anaesthesia · doi:10.1093/bja/84.5.663 · Cited by 206
Why is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185 · cited 1×
“Future work will need to address the under-representation of sub-populations in the training data 28 and overcome the impact of potential confounding factors related to hospital processes 29 .”
Relational recurrent neural networksPreprint
Santoro, A., Faulkner, R., Raposo, D. C., Rae, J. W., Chrzanowski, M., Weber, T., Wierstra, D., Vinyals, O., Pascanu, R., & Lillicrap, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01822 · Cited by 182
Self-citationMohamadlou, H., Lynn-Palevsky, A., Barton, C., Chettipally, U. K., Shieh, L. Y., Calvert, J. S., Saber, N., & Das, R. (2018) · Canadian Journal of Kidney Health and Disease · doi:10.1177/2054358118776326 · Cited by 168
Capacity and Trainability in Recurrent Neural NetworksPreprint
Collins, J., Sohl‐Dickstein, J., & Sussillo, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09913 · Cited by 149
Park, S., Baek, S. H., Ahn, S., Lee, K.-H., Hwang, H. Y., Ryu, J., Ahn, S. Y., Chin, H. J., Na, K. Y., Chae, D., et al. (2017) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2017.06.008 · Cited by 134
Perotte, A., Ranganath, R., Hirsch, J. S., Blei, D., & Elhadad, N. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv024 · Cited by 128
Learning to Detect Sepsis with a Multitask Gaussian Process RNN ClassifierPreprint
Futoma, J., Hariharan, S., & Heller, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04152 · Cited by 124
Lachance, P., Villeneuve, P.-M., Rewa, O. G., Wilson, F. P., Selby, N. M., Featherstone, R. M., & Bagshaw, S. M. (2016) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfw424 · Cited by 115
Development of a Multicenter Ward–Based AKI Prediction Model
Koyner, J. L., Adhikari, R., Edelson, D. P., & Churpek, M. M. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.00280116 · Cited by 113
Personalized Risk Scoring for Critical Care Prognosis Using Mixtures of Gaussian Processes
Alaa, A. M., Yoon, J., Hu, S. B., & van der Schaar, M. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2017.2698602 · Cited by 74
NCEPOD report on acute kidney injury—must do better
MacLeod, A. M. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61843-2 · Cited by 68
Dovgan, E., Gradišek, A., Luštrek, M., Uddin, M., Nursetyo, A. A., Annavarajula, S. K., Li, Y., & Syed-Abdul, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233976 · Cited by 56
Patient risk stratification with time-varying parameters: a multitask learning approach
Wiens, J., Guttag, J. V., & Horvitz, E. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 54
Predicting Inpatient Acute Kidney Injury over Different Time Horizons: How Early and Accurate?
Cheng, P., Waitman, L. R., Hu, Y. Q., & Liu, M. (2017) · PubMed Central · Cited by 51
Temporal Convolutional Neural Networks for Diagnosis from Lab TestsPreprint
Razavian, N., & Sontag, D. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07938 · Cited by 49
Will AI Replace Ophthalmologists?
Korot, E., Wagner, S. K., Faes, L., Liu, X., Huemer, J. C., Ferraz, D., Keane, P. A., & Balaskas, K. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.2 · Cited by 39
Deng, F., Peng, M., Li, J., Chen, Y., Zhang, B., & Zhao, S. (2020) · Renal Failure · doi:10.1080/0886022x.2020.1761832 · Cited by 39
Artificial intelligence and computer simulation models in critical illness
Lal, A., Pinevich, Y., Gajic, O., Herasevich, V., & Pickering, B. (2020) · World Journal of Critical Care Medicine · doi:10.5492/wjccm.v9.i2.13 · Cited by 34
The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data
Ding, D. Y., Simpson, C., Pfohl, S., Kale, D., Jung, K., & Shah, N. H. (2018) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813279827_0003 · Cited by 28
Soleimani, H., Subbaswamy, A., & Saria, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02038 · Cited by 23
Disease-Atlas: Navigating Disease Trajectories using Deep Learning
Lim, B. Y., & van der Schaar, M. (2018) · Machine Learning for Healthcare Conference · Cited by 19
Personalized Antibiograms: Machine Learning for Precision Selection of Empiric Antibiotics.
Corbin, C. K., Medford, R. J., Osei, K., & Chen, J. H. (2020) · PubMed Central · Cited by 19
Personalized Risk Scoring for Critical Care Patients using Mixtures of Gaussian Process ExpertsPreprint
Alaa, A. M., Yoon, J., Hu, S. B., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.00959 · Cited by 12
A Self-Correcting Deep Learning Approach to Predict Acute Conditions in Critical CarePreprint
Pan, Z., Du, H. Y., Ngiam, K. Y., Wang, F., Shum, P., & Feng, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04364 · Cited by 10
Matsuno, K., Tanaka, S., Hashimoto, T., Nakamichi, H., & Komura, E. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfx149.sp428 · Cited by 9
Telenti, A., Steinhubl, S. R., & Topol, E. J. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30538-5 · Cited by 7
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“To compare uncalibrated predictions to recalibrated ones we used the Brier score 51 and reliability plots 52 .”
Research works printed but not linked: 3
- Schulam, P. & Saria, S. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 1697–1708 (2017).
- Lei, T. & Zhang, Y. Training RNNs as fast as CNNs. Preprint at https://arxiv.org/abs/1709.02755v1 (2017). · arXiv
- Caruana, R., Baluja, S. & Mitchell, T. in Advances in Neural Information Processing Systems (eds Mozer, M. et al.) 959–965 (1996).
Cited by
Sorted by citations
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Self-citationAcosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
Improving the accuracy of medical diagnosis with causal machine learning
Richens, J. G., Lee, C. M., & Johri, S. (2020) · Nature Communications · doi:10.1038/s41467-020-17419-7 · Cited by 598
Metadata
- Authors
- Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., Connell, A., Hughes, C. O., Karthikesalingam, A., Cornebise, J., Montgomery, H., Rees, G., Laing, C., Baker, C. R., Peterson, K., Reeves, R., Hassabis, D., King, D., Suleyman, M., Back, T., Nielson, C., Ledsam, J. R., & Mohamed, S.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- University College London (GB)
- University College Hospital (GB)
- United States Department of Veterans Affairs (US)
- University of Utah (US)
- University of Nevada, Reno (US)
- Corresponding author
- Tomašev, Nenad
- Ledsam, Joseph R.
- ORCID
- Venue
- Nature
- Published
- 2019-07-31
- Volume
- 572
- Issue
- 7767
- Pages
- 116-119
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC6722431
- OpenAlex
- W2964696298
- Cited by
- 1,086
- References
- 60
- Citations by year
- 2026: 123 · 2025: 149 · 2024: 122 · 2023: 150 · 2022: 154 · 2021: 169 · 2020: 180 · 2019: 39
- FWCI
- 61.95
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Institutes of Health (VA999999)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- acute kidney injury, deep learning, electronic health records, clinical deterioration, dialysis requirement, confidence estimation, false alarms, clinical decision support
- MeSH terms
- Adolescent, Adult, Aged, Aged, 80 and over, Computer Simulation, False Positive Reactions, Female, Humans, Male, Middle Aged, ROC Curve, Risk Assessment, Clinical Laboratory Techniques, Pulmonary Disease, Chronic Obstructive, Uncertainty, Young Adult, Acute Kidney Injury, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
