Time-dependent prediction and evaluation of variable importance using superlearning in high-dimensional clinical data.
Open accessAbstract
BACKGROUND: Prediction of outcome after injury is fraught with uncertainty and statistically beset by misspecified models. Single-time point regression only gives prediction and inference at one time, of dubious value for continuous prediction of ongoing bleeding. New statistical machine learning techniques such as SuperLearner (SL) exist to make superior prediction at iterative time points while evaluating the changing relative importance of each measured variable on an outcome. This then can provide continuously changing prediction of outcome and evaluation of which clinical variables likely drive a particular outcome. METHODS: PROMMTT data were evaluated using both naive (standard stepwise logistic regression) and SL techniques to develop a time-dependent prediction of future mortality within discrete time intervals. We avoided both underfitting and overfitting using cross validation to select an optimal combination of predictors among candidate predictors/machine learning algorithms. SL was also used to produce interval-specific robust measures of variable importance measures (VIM resulting in an ordered list of variables, by time point) that have the strongest impact on future mortality. RESULTS: Nine hundred eighty patients had complete clinical and outcome data and were included in the analysis. The prediction of ongoing transfusion with SL was superior to the naive approach for all time intervals (correlations of cross-validated predictions with the outcome were 0.819, 0.789, 0.792 for time intervals 30-90, 90-180, 180-360, >360 minutes). The estimated VIM of mortality also changed significantly at each time point. CONCLUSION: The SL technique for prediction of outcome from a complex dynamic multivariate data set is superior at each time interval to standard models. In addition, the SL VIM at each time point provides insight into the time-specific drivers of future outcome, patient trajectory, and targets for clinical intervention. Thus, this automated approach mimics clinical practice, changing form and content through time to optimize the accuracy of the prognosis based on the evolving trajectory of the patient.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Super Learner ranks candidate algorithms by performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.
“To capitalize on this phenomenon, Super Learner ranks a set of candidate algorithms20,28,38,39 by their performance and applies an optimal weight to each, creating ensemble algorithms that can accurately predict transfusion requirements and mortality among trauma patients.20,28,38,39”
From Artificial Intelligence and Surgical Decision-making. (2020) · cites this paper as [28]
The Methods (page unavailable) say SuperLearner uses cross-validation to combine candidate algorithms and select the best-performing approach. The Abstract Results and Results > Prediction report predictions of ongoing transfusion and death in trauma patients. Our copy does not specify that SuperLearner applies an optimal weight to each candidate algorithm. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Gentleman R: R: A language for data analysis and graphics
Ihaka, R., & Gentleman, R. (1996) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1996.10474713 · Cited by 11,747 · see citing works
Bayesian inference for causal effects: The role of randomization
Rubin, D. B. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344064 · Cited by 2,503 · see citing works
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 2,115 · see citing works
van der Laan, M. J., Polley, E., & Hubbard, A. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,472 · see citing works
Self-citationA WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION FOR LOGISTIC AND OTHER REGRESSION MODELS
Gelman, A., Jakulin, A., & Grazia, M. (2009) · Cited by 1,352 · see citing works
Causes of Death in U.S. Special Operations Forces in the Global War on Terrorism
Holcomb, J. B., McMullin, N. R., Pearse, L. A., Caruso, J., Wade, C. E., Oetjen-Gerdes, L., Champion, H. R., Lawnick, M., Farr, W. D., Rodriguez, S., et al. (2007) · Annals of Surgery · doi:10.1097/01.sla.0000259433.03754.98 · Cited by 711 · see citing works
Self-citationEarly Prediction of Massive Transfusion in Trauma: Simple as ABC (Assessment of Blood Consumption)?
Nuñez, T. C., Voskresensky, I. V., Dossett, L. A., Shinall, R., Dutton, W. D., & Cotton, B. A. (2009) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e3181961c35 · Cited by 577 · see citing works
Self-citationFIBTEM provides early prediction of massive transfusion in trauma
Schöchl, H., Cotton, B. A., Inaba, K., Nienaber, U., Fischer, H., Voelckel, W., & Solomon, C. (2011) · Critical Care · doi:10.1186/cc10539 · Cited by 314 · see citing works
Self-citationHess, J. R., Holcomb, J. B., & Hoyt, D. B. (2006) · Transfusion · doi:10.1111/j.1537-2995.2006.00816.x · Cited by 243 · see citing works
Self-citationTargeted Maximum Likelihood Based Causal Inference: Part II
van der Laan, M. J. (2010) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1241 · Cited by 239 · see citing works
Population Intervention Causal Effects Based on Stochastic Interventions
Muñoz, I. D., & van der Laan, M. J. (2011) · Biometrics · doi:10.1111/j.1541-0420.2011.01685.x · Cited by 167 · see citing works
Rahbar, M. H., Fox, E. E., del Junco, D. J., Cotton, B. A., Podbielski, J. M., Matijevic, N., Cohen, M. J. M., Schreiber, M. A., Zhang, J., Mirhaji, P., et al. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.09.019 · Cited by 55 · see citing works
Self-citationComparison of massive blood transfusion predictive models in the rural setting
Krumrei, N. J., Park, M. S., Cotton, B. A., & Zielinski, M. D. (2012) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e318240507b · Cited by 49 · see citing works
Self-citationLesko, M. M., Jenks, T., O’Brien, S. J., Childs, C., Bouamra, O., Woodford, M., & Lecky, F. E. (2012) · Journal of Neurotrauma · doi:10.1089/neu.2012.2438 · Cited by 39 · see citing works
Novel representation of physiologic states during critical illness and recovery
Buchman, T. G. (2010) · Critical Care · doi:10.1186/cc8868 · Cited by 30 · see citing works
A model for predicting primary blast lung injury
MacFadden, L. N., Chan, P. C., Ho, K. H.-H., & Stuhmiller, J. H. (2012) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e31825c1536 · Cited by 26 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [11]van der Laan M Rose S Targeted learning: causal inference for observational and experimental data 2011 New York Springer
- [15]van der Laan MJ Rubin DB Targeted maximum likelihood learning Int J Biostat 2006 2 PMC3126670 21969992
- [19]Pearl J Causality 2000 Cambridge Cambridge University Press
Cited by
Sorted by citations
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 611 · see citing works
Trauma‐induced coagulopathy: The past, present, and future
Kornblith, L. Z., Moore, H. B., & Cohen, M. J. M. (2019) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.14450 · Cited by 292 · see citing works
Self-citationMachine Learning for Predicting Outcomes in Trauma
Liu, N. T., & Salinas, J. (2017) · Shock · doi:10.1097/shk.0000000000000898 · Cited by 114 · see citing works
Can Hyperparameter Tuning Improve the Performance of a Super Learner?
Wong, J., Manderson, T., Abrahamowicz, M., Buckeridge, D., & Tamblyn, R. (2019) · Epidemiology · doi:10.1097/ede.0000000000001027 · Cited by 50 · see citing works
Comparison of massive blood transfusion predictive models in the rural setting
Krumrei, N. J., Park, M. S., Cotton, B. A., & Zielinski, M. D. (2012) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e318240507b · Cited by 49 · see citing works
Self-citationChristie, S. A., Conroy, A. S., Callcut, R. A., Hubbard, A., & Cohen, M. J. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213836 · Cited by 46 · see citing works
Self-citationMachine Learning in the Prediction of Trauma Outcomes: A Systematic Review
Zhang, T., Nikouline, A., Lightfoot, D., & Nolan, B. (2022) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.05.011 · Cited by 42 · see citing works
Shete, P. B., Ravindran, R. K., Chang, E. G., Worodria, W. O., Chaisson, L. H., Andama, A. O., Davis, J. L., Luciw, P. A., Huang, L., Khan, I., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0180122 · Cited by 35 · see citing works
Christie, S. A., Hubbard, A. E., Callcut, R. A., Hameed, M., Dissak-Delon, F. N., Mekolo, D., Saidou, A., Mefire, A. C., Nsongoo, P., Dicker, R. A., et al. (2018) · Journal of Trauma and Acute Care Surgery · doi:10.1097/ta.0000000000002044 · Cited by 33 · see citing works
Self-citationGruber, S., Krakower, D., Menchaca, J., Hsu, K., Hawrusik, R., Maro, J. C., Cocoros, N., Kruskal, B., Wilson, I., Mayer, K., et al. (2020) · Statistics in Medicine · doi:10.1002/sim.8591 · Cited by 26 · see citing works
Baur, D., Gehlen, T., Scherer, J. N., Back, D. A., Tsitsilonis, S., Kabir, K., & Osterhoff, G. (2022) · Frontiers in Surgery · doi:10.3389/fsurg.2022.924810 · Cited by 24 · see citing works
Scalable Ensemble Learning and Computationally Efficient Variance Estimation
LeDell, E. (2015) · eScholarship (California Digital Library) · Cited by 21 · see citing works
Macalinao, M. L., Fornace, K., Reyes, R. A., Hall, T., Bareng, A. P. N., Adams, J. H., Huon, C., Chitnis, C. E., Luchavez, J., Tetteh, K. K. A., et al. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100792 · Cited by 21 · see citing works
Farr, K., Ravindran, R. K., Strnad, L. C., Chang, E. G., Chaisson, L. H., Yoon, C., Worodria, W. O., Andama, A. O., Ayakaka, I., Nalwanga, P. B., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0206119 · Cited by 20 · see citing works
Proteomic profiles by multiplex microsphere suspension array
Krishnan, V. V., Selvan, S. R., Parameswaran, N., Venkateswaran, N., Luciw, P. A., & Venkateswaran, K. S. (2018) · Journal of Immunological Methods · doi:10.1016/j.jim.2018.07.002 · Cited by 17 · see citing works
Health system measurement: Harnessing machine learning to advance global health
Leslie, H. H., Zhou, X., Spiegelman, D. L., & Kruk, M. E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0204958 · Cited by 14 · see citing works
Data-Adaptive Target Parameters
Hubbard, A., Kennedy, C. J., & van der Laan, M. J. (2018) · Springer series in statistics · doi:10.1007/978-3-319-65304-4_9 · Cited by 13 · see citing works
Self-citationPirracchio, R., Yue, J. K., Manley, G. T., van der Laan, M. J., Hubbard, A., & Lingsma, H. F. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280215627335 · Cited by 13 · see citing works
Self-citationPrehospital factors predict outcomes in pediatric trauma: A principal component analysis
Ting, T., Wakeman, D. S., Arca, M. J., & Wilson, N. A. (2022) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000003680 · Cited by 7 · see citing works
Nugent, J. R., & Balzer, L. B. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad005 · Cited by 7 · see citing works
Targeted Learning in Healthcare Research
Gruber, S. (2015) · Big Data · doi:10.1089/big.2015.0025 · Cited by 7 · see citing works
Machine learning methods for predicting traumatic injuries outcomes
Almaghrabi, F. (2021) · Research Explorer (The University of Manchester) · Cited by 6 · see citing works
Change talk subtypes as predictors of alcohol use following brief motivational intervention.
Kahler, C. W., Janssen, T., Gruber, S., Howe, C. J., Laws, M. B., Walthers, J., Magill, M., Mastroleo, N. R., & Monti, P. M. (2022) · Psychology of Addictive Behaviors · doi:10.1037/adb0000898 · Cited by 3 · see citing works
Rajkumar, E., Nguyen, K. M., Radic, S., Paa, J., & Geng, Q. (2023) · JMIR Formative Research · doi:10.2196/41725 · Cited by 4 · see citing works
Translational approaches to coagulopathy after trauma: Towards targeted treatment
Cohen, M. J. M. (2017) · PLoS Medicine · doi:10.1371/journal.pmed.1002359 · Cited by 4 · see citing works
Self-citationComputers in White Coats: How to Devise Useful Clinical Decision Support Software*
Moorman, J. R., Lake, D. E., & Moss, T. J. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001781 · Cited by 2 · see citing works
Research Questions in Data Science
Rose, S., & van der Laan, M. V. (2018) · Springer series in statistics · doi:10.1007/978-3-319-65304-4_1
Data-Adaptive Estimation in Cluster Randomized Trials
Balzer, L., van der Laan, M. V., & Petersen, M. (2018) · Springer series in statistics · doi:10.1007/978-3-319-65304-4_13
Ritter, S. (2013) · eScholarship (California Digital Library)
Developing Methods to Predict Health Outcomes in Trauma Patients
Hubbard, A., & van der Laan, M. J. (2020) · doi:10.25302/1.2020.me.130602735
Self-citationNugent, J. R., & Balzer, L. B. (2021) · arXiv (Cornell University)
Park, E., Lee, H., Oh, E. J., Tham, T., & Ahn, S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.11.731237
- Authors
- Hubbard, A., Munoz, I. D., Decker, A., Holcomb, J. B., Schreiber, M. A., Bulger, E. M., Brasel, K. J., Fox, E. E., del Junco, D. J., Wade, C. E., Rahbar, M. H., Cotton, B. A., Phelan, H. A., Myers, J. G., Alarcon, L. H., Muskat, P., & Cohen, M. J.
- Affiliations
- University of California, Berkeley (US)
- The University of Texas Health Science Center at Houston (US)
- Oregon Health & Science University (US)
- University of Washington (US)
- Medical College of Wisconsin (US)
- The University of Texas Southwestern Medical Center (US)
- The University of Texas at San Antonio Health Science Center (US)
- University of Pittsburgh (US)
- University of Cincinnati (US)
- University of California, San Francisco (US)
- ORCID
- Venue
- Journal of Trauma and Acute Care Surgery
- Published
- 2013-07
- Volume
- 75
- Issue
- Supplement 1
- Pages
- S53-S60
- ISSN
- 2163-0755 (ISSN-L)
- PMC
- PMC3744063
- OpenAlex
- W2011534852
- Cited by
- 32
- References
- 17
- Citations by year
- 2026: 1 · 2023: 3 · 2022: 4 · 2021: 2 · 2020: 2 · 2019: 4 · 2018: 7 · 2017: 3 · 2016: 2 · 2015: 2 · 2013: 1 · 2012: 1
- FWCI
- 1.47
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- The Journal of Trauma: Injury, Infection, and Critical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of General Medical Sciences (K08GM085689)
- National Center for Research Resources (UL1RR024148)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- super learner, temporal prediction, variable importance measures, mortality prediction, blood transfusion prediction, trauma patients, ensemble learning, cross-validation
- MeSH terms
- Adult, Algorithms, Artificial Intelligence, Blood Transfusion, Female, Hemorrhage, Humans, Male, Middle Aged, Predictive Value of Tests, Prospective Studies, Resuscitation, Time Factors, Trauma Centers, United States, Wounds and Injuries, Models, Statistical, Injury Severity Score, Survival Analysis, Treatment Outcome, Hospital Mortality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
