Machine learning for real-time aggregated prediction of hospital admission for emergency patients

May combine two claims.

Logistic Regression with Lasso regularisation gave performance almost as good as XGBoost, with increased regularisation giving less good performance.

“Logistic Regression with Lasso regularisation gave performance almost as good as XGBoost, with increased regularisation (and as a consequence sparser regression models) giving less good performance (Fig. 4).”

• Also stated in: § Discussion · plain

  “We found that XGBoost outperformed Random Forest and that LR optimised with Lasso regularisation was a close second.”

May combine two claims.

Triage scores are important up to 60 minutes, and the National Early Warning Score (NEWS) remains important up to 240 minutes.

“Triage scores are important up to 60 minutes, and the National Early Warning Score (NEWS) remains important up to 240 minutes.”

May combine two claims.

After Step 5, concurrence remains good, although especially for an eight-hour prediction window the predicted distributions underestimate slightly the number of admissions within the prediction window.

“After Step 5, concurrence remains good, although (especially for an eight-hour prediction window) the predicted distributions underestimate slightly the number of admissions within the prediction window, suggesting that patients were taking less time to be admitted than predicted.”

May combine two claims.

The models achieved lowest log loss when presented with patients with elapsed times of between 30 and 120 minutes, and the best AUROC of 0.90 between 90 and 180 minutes.

“The models achieved lowest log loss when presented with patients with elapsed times of between 30 and 120 minutes, and the best Area Under the Receiving Operating Curve (AUROC) of 0.90 between 90 and 180 minutes.”

May combine two claims.

Patient-level probabilities of admission were aggregated to forecast the number of admissions among current ED patients and, incorporating patients yet to arrive, total emergency admissions within specified time-windows.

“Patient-level probabilities of admission were aggregated to forecast the number of admissions among current ED patients and, incorporating patients yet to arrive, total emergency admissions within specified time-windows.”

• AssumptionRests on an assumption: The admission outcomes of individual patients in the ED are independent, so their probabilities can be combined into a probability distribution for the total number of admissions.

Probably not a substantial claim: may not pass S3 (load-bearing).

Both Random Forest and Logistic Regression models drew on similar feature sets to XGBoost.

“Both Random Forrest and Logistic Regression models drew on similar feature sets to XGBoost (Supplementary Note 8).”

May combine two claims.

All ML models are well calibrated, up to the final two models which related to a very small subset of visits where patients remain in the ED after 8 hours.

“All models are well calibrated, up to the final two models which related to a very small subset of visits where patients remain in the ED after 8 hours.”

May combine two claims.

The prediction pipeline gave a mean absolute error (MAE) of 4.0 admissions (mean percentage error of 17%) versus 6.5 (32%) for a benchmark metric, and outperformed the conventional six-week rolling average benchmark.

“The pipeline gave a mean absolute error (MAE) of 4.0 admissions (mean percentage error of 17%) versus 6.5 (32%) for a benchmark metric.”

• Also stated in: § Results for steps 3–7: evaluation of aggregated predictions · plain

  “The prediction pipeline underestimated the number of admissions within the 8-hour window but performed better than the benchmark (MAE of 4.0 admissions with MPE of 17%, compared with 6.5 and 32% for the benchmark).”

• Also stated in: § Discussion · plain

  “The predictions outperformed the conventional six-week rolling average benchmark.”

• AssumptionRests on an assumption: The test set of 97 days from 13 December 2019 to 18 March 2020 is representative of the operational period for which the model is intended.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The seven-step pipeline needed one adjustment to accommodate the greater variation in both patients and operational conditions after the outbreak of Covid; this was to refine the survival curve used in Step 4 to draw only on rolling data from the past six weeks.

“The seven-step pipeline needed one adjustment to accommodate the greater variation in both patients and operational conditions after the outbreak of Covid; this was to refine the survival curve used in Step 4 to draw only on rolling data from the past six weeks.”

• AssumptionRests on an assumption: The Covid-19 pandemic caused greater variation in both patients and operational conditions.

May combine two claims.

Numbers of ED presentations fell dramatically in late March 2020 and patients were processed quickly.

“Numbers of presentations fell dramatically in late March 2020 (see Supplementary Fig. 3) and patients were processed quickly.”

May combine two claims.

After a quiet period in April to June 2020, presentations gradually climbed, reaching pre-pandemic levels by mid-2021 and EDs have become progressively busier and slower to admit patients.

“After a quiet period in April to June 2020, presentations gradually climbed, reaching pre-pandemic levels by mid-2021 and EDs have become progressively busier and slower to admit patients.”

Classified as probably not a substantial claim.

The two years in this study were very different, affected by organisation changes and the impact of Covid-19 on health services.

“in fact the two years in this study were very different, affected by organisation changes and the impact of Covid-19 on health services.”

May combine two claims.

Patterns of feature importance and model performance were largely consistent across three different datasets, confirming that the ML models are relatively robust to changes brought about by the pandemic.

“Patterns of feature importance (Supplementary Fig. 11) and model performance (Supplementary Fig. 12) were largely consistent across the three datasets.”

• Also stated in: § Discussion · plain

  “Patterns of feature importance and model performance were largely consistent across three different datasets, confirming that the ML models are relatively robust to changes brought about by the pandemic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After filtering, there were 213,985 visits.

“After filtering, there were 213,985 visits.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The benefits of the study are hard to isolate from the contribution of other workflows and initiatives in a complex and changing environment.

“but its benefits are hard to isolate from the contribution of other workflows and initiatives in a complex and changing environment.”

May combine two claims.

Using logistic regression, Barak-Corren et al. achieved an AUROC of 0.82 in their 10 minute model, compared with 0.85 at 15 minutes here without chief complaint; they achieved an AUROC of 0.83 after 120 minutes, compared with 0.90 here.

“Using logistic regression (LR), Barak-Corren et al.18 achieved an AUROC of 0.82 in their 10 minute model (augmented by free-text chief complaint data that was retrospectively encoded), compared with 0.85 at 15 minutes here without chief complaint; they achieved an AUROC of 0.83 after 120 minutes, compared with 0.90 here.”

• cites [18] Barak-Corren, Y., Israelit, S. H. & Reis, B. Y. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg. Med. J. 34, 308–314 (2017). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The costs of implementation were small as the hospital had already invested in the infrastructure to utilise data from the EHR, and no data entry was required by clinical staff in real-time.

“The costs of implementation were small as the hospital had already invested in the infrastructure to utilise data from the EHR, and no data entry was required by clinical staff in real-time.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A post-hoc change to the step that uses survival curves was made for the period following the Covid outbreak to address fluctuations in the flow of patients through the ED after the pandemic began.

“For example, a post-hoc change to the step that uses survival curves was made for the period following the Covid outbreak to address fluctuations in the flow of patients through the ED after the pandemic began.”

May combine two claims.

Using XGBoost, Barak-Corren et al. achieved AUROC of 0.87-0.93 with a 60 minute model, compared with 0.89 here, and Hong et al. achieved 0.87 with triage data.

“Using XGBoost, Barak-Corren et al.31 achieved AUROC of 0.87–0.93 with a 60 minute model, compared with 0.89 here, and Hong et al9 achieved 0.87 with triage data.”

• cites [31] Barak-Corren, Y. et al. Prediction across healthcare settings: a case study in predicting emergency department disposition. Npj Digit. Med. 4, 1–7 (2021). sourceNot checked yet

  Not checked in the free run.

• cites [9] Hong, W. S., Haimovich, A. D. & Taylor, R. A. Predicting hospital admission at emergency department triage using machine learning. PLoS ONE 13, 1–13 (2018). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Comparison with the commonly used six-week rolling average was not straightforward, as this metric is for a 24 hour prediction window from midnight.

“Comparison with the commonly used six-week rolling average was not straightforward, as this metric is for a 24 hour prediction window from midnight.”

Classified as probably not a substantial claim.

When making predictions for admissions within a time-window after the prediction is made, projections must allow for the number of patients not in the ED at the prediction time who will arrive and be admitted within the window.

“Also, when making predictions for admissions within a time-window after the prediction is made, projections must allow for the number of patients not in the ED at the prediction time who will arrive and be admitted within the window28.”

• cites [28] Utley, M., Gallivan, S., Treasure, T. & Valencia, O. Analytical methods for calculating the capacity required to operate an effective booked admissions policy for elective inpatient services. Health Care Manag. Sci. 6, 97–104 (2003). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Barak-Corren et al.'s study built progressive datasets from historical data, each intended to reflect the data usually available at 10, 60 and 120 minutes after presentation to the ED.

“They built progressive datasets from historical data, each intended to reflect the data usually available at 10, 60 and 120 minutes after presentation to the ED.”

• cites [18] Barak-Corren, Y., Israelit, S. H. & Reis, B. Y. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg. Med. J. 34, 308–314 (2017). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Such methods do not take account of the stochastic nature of ED arrivals and cannot be adapted to reflect the case mix of people in the ED at a given point in time.

“However, such methods do not take account of the stochastic nature of ED arrivals13 and cannot be adapted to reflect the case mix of people in the ED at a given point in time.”

• cites [13] Demeester, P., Souffriau, W., De Causmaecker, P. & Vanden Berghe, G. A hybrid tabu search algorithm for automatically assigning patients to beds. Artif. Intell. Med. 48, 61–70 (2010). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

These clinical contexts have patient observations taken with high frequency, whereas in the ED the frequency and type of data collection varies greatly from patient to patient.

“These are contexts where patient observations are taken with high frequency whereas the frequency of data collection and the type of data collected varies greatly from patient to patient in the ED.”

Probably not a substantial claim: may not pass S2 (checkable).

Real-time operational models need to cover the 'last mile' of AI deployment, meaning that applications that generate predictions can run end-to-end without human intervention.

“Real-time operational models also need to cover the ‘last mile’ of AI deployment; this means that the applications that generate predictions can run end-to-end without human intervention.”

• cites [29] Seneviratne, M. G., Shah, N. H. & Chu, L. Bridging the implementation gap of machine learning in healthcare. BMJ Innov. 6, (2020). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Models for real-time prediction have been trained in clinical contexts such as circulatory failure in critical care and post-operative complications.

“Models for real-time prediction have been trained in clinical contexts such as circulatory failure in critical care24 and post-operative complications25,26.”

• cites [24] Hyland, S. L. et al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nat. Med. 26, 364–373 (2020). sourceNot checked yet

  Not checked in the free run.

• cites [25] Fritz, B. A. et al. Using machine learning techniques to develop forecasting algorithms for postoperative complications: protocol for a retrospective study. BMJ Open 8, e020124 (2018). sourceNot checked yet

  Not checked in the free run.

• cites [26] Meyer, A. et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir. Med. 6, 905–914 (2018). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hong et al. showed that ML algorithms like gradient-boosted trees and deep neural networks, applied to a large EHR dataset of 972 variables, improved predictive performance.

“Hong et al.9 showed that ML algorithms like gradient-boosted trees and deep neural networks, applied to a large EHR dataset of 972 variables, improved predictive performance.”

• cites [9] Hong, W. S., Haimovich, A. D. & Taylor, R. A. Predicting hospital admission at emergency department triage using machine learning. PLoS ONE 13, 1–13 (2018). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

As time goes on, the more straightforward discharges and admissions are made, the number of training examples diminishes and the case mix includes a higher proportion of more clinically complex cases which are harder to predict.

“As time goes on, the more straightforward discharges and admissions are made, the number of training examples diminishes (see Fig. 2b) and the case mix includes a higher proportion of more clinically complex cases which are harder to predict.”

Probably not a substantial claim: may not pass S2 (checkable).

If models are to be used operationally, their performance needs to be sustained over time as care provision, patient characteristics and the systems used to capture data evolve.

“If models are to be used operationally, their performance needs to be sustained over time as care provision, patient characteristics and the systems used to capture data evolve23.”

• cites [23] Nestor, B. et al. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Preprint at https://arxiv.org/abs/1908.00690 (2019).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Heterogeneous data profiles in the ED are themselves indicative of likelihood of admission.

“These heterogeneous data profiles are themselves indicative of likelihood of admission.”

Probably not a substantial claim: may not pass S3 (load-bearing).

El-Bouri et al. were able to predict which medical specialty patients would be admitted to by including data on lab test results and procedures.

“By including data on lab test results and procedures, El-Bouri et al.20 were able to predict which medical specialty patients would be admitted to.”

• cites [20] El-Bouri, R., Eyre, D. W., Watkinson, P., Zhu, T. & Clifton, D. A. Hospital admission location prediction via deep interpretable networks for the year-round improvement of emergency patient care. IEEE J. Biomed. Health Inform. 25, 289–300 (2021). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Barak-Corren et al. showed that the later datasets offered better predictions than at 10 minutes.

“they were able to show that the later datasets offered better predictions than at 10 minutes.”

• cites [18] Barak-Corren, Y., Israelit, S. H. & Reis, B. Y. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg. Med. J. 34, 308–314 (2017). sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

In hospitals with EHRs, where staff are recording patient data at the point of care, there is an opportunity to use EHR data to generate short-horizon predictions of bed demand.

“In hospitals with EHRs, where staff are recording patient data at the point of care, there is an opportunity to use EHR data to generate short-horizon predictions of bed demand.”

Classified as probably not a substantial claim.

Machine learning is attractive for such predictions because its aggregation of weak predictors may create a strong prediction model.

“Maching Learning (ML) is attractive for such predictions because its aggregation of weak predictors may create a strong prediction model2.”

• cites [2] Pianykh, O. S. et al. Improving healthcare operations management with machine learning. Nat. Mach. Intell. 2, 266–273 (2020). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The slight underestimation after Step 5 suggests that patients were taking less time to be admitted than predicted.

“After Step 5, concurrence remains good, although (especially for an eight-hour prediction window) the predicted distributions underestimate slightly the number of admissions within the prediction window, suggesting that patients were taking less time to be admitted than predicted.”

May combine two claims.

Building a model for implementation involves additional challenges to those encountered when simply optimising the technical performance of a prediction model, including preparing training examples of incomplete visits from historic data, making decisions about temporal framing, and planning for drift in model performance over time.

“Building a model for implementation involves several additional challenges to those encountered when simply optimising the technical performance of a prediction model. These include preparing training examples of incomplete visits from historic data in which visits have been completed21, making decisions about the temporal framing of the model (for example, at what point in the visit to check if t”

• cites [21] Kanter, J. M., Gillespie, O. & Veeramachaneni, K. Label, segment, featurize: a cross domain framework for prediction engineering. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) 430–439. https://doi.org/10.1109/DSAA.2016.54 (2016). sourceNot checked yet

  Not checked in the free run.

• cites [22] Lauritsen, S. M. et al. The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards. Npj Digit. Med. 4, 1–12 (2021). sourceNot checked yet

  Not checked in the free run.

• cites [23] Nestor, B. et al. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Preprint at https://arxiv.org/abs/1908.00690 (2019).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

This last mile is the most neglected, leading to calls for a delivery science for AI, in which AI is viewed as an enabling component within an operational workflow, rather than an end in itself.

“This last mile is the most neglected29, leading to calls for a delivery science for AI, in which AI is viewed as an enabling component within an operational workflow, rather than an end in itself30.”

• cites [29] Seneviratne, M. G., Shah, N. H. & Chu, L. Bridging the implementation gap of machine learning in healthcare. BMJ Innov. 6, (2020). sourceNot checked yet

  Not checked in the free run.

• cites [30] Li, R. C., Asch, S. M. & Shah, N. H. Developing a delivery science for artificial intelligence in healthcare. Npj Digit. Med. 3, 1–3 (2020). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

One study in emergency medicine derived an expected number of admissions among a roomful of patients in ED by summing their individual probabilities of admission, but there was no presentation of the uncertainty of their point estimates.

“One study in emergency medicine derived an expected number of admissions among a roomful of patients in ED by summing their individual probabilities of admission27, but there was no presentation of the uncertainty of their point estimates.”

• cites [27] Peck, J. S., Benneyan, J. C., Nightingale, D. J. & Gaehde, S. A. Predicting emergency department inpatient admissions to improve same-day patient flow. Acad. Emerg. Med 19, E1045–E1054 (2012). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Other settings without the upfront investment in infrastructure might be able to achieve similar results with fewer data points and simpler models, as suggested by our LR results.

“Other settings without the upfront investment in infrastructure might be able to achieve similar results with fewer data points and simpler models, as suggested by our LR results”

May combine two claims.

Aside from those based on location, the risk factors for admission found here are familiar in ED practice and therefore likely to be common to other sites.

“But aside from those based on location, the risk factors for admission found here are familiar in ED practice and therefore likely to be common to other sites31.”

• cites [31] Barak-Corren, Y. et al. Prediction across healthcare settings: a case study in predicting emergency department disposition. Npj Digit. Med. 4, 1–7 (2021). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The full set of variables was available for inclusion in the regression models.

“That said, we note that that the full set of variables was available for inclusion in the regression models.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study was deemed exempt from NHS Research Ethics Committee review as there is no change to treatment or services or any study randomisation of patients into different treatment groups.

“The study was deemed exempt from NHS Research Ethics Committee review as there is no change to treatment or services or any study randomisation of patients into different treatment groups.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

It may be possible to implement some predictive models within an EHR.

“and it may be possible to implement some predictive models within an EHR.”

Classified as probably not a substantial claim. May combine two claims.

This work draws attention to some challenges that make technically high-performing systems perform poorly for their intended use, including model drift, and considers how to address them.

“This work draws attention to some challenges that make technically high-performing systems perform poorly for their intended use, including model drift, and considers how to address them.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Here we demonstrate an application of ML in that last mile, providing an example of how ML for healthcare will need to be delivered if it is to become a dependable and reliable tool.

“Here we demonstrate an application of ML in that last mile, providing an example of how ML for healthcare will need to be delivered if it is to become a dependable and reliable tool.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Crowding in EDs is an outcome of system-wide issues downstream such as bottlenecks and capacity constraints.

“Ultimately of course, crowding in EDs is an outcome of system-wide issues downstream such as bottlenecks and capacity constraints6,37.”

• cites [6] McKenna, P. et al. Emergency department and hospital crowding: causes, consequences, and cures. Clin. Exp. Emerg. Med. 6, 189–195 (2019). sourceNot checked yet

  Not checked in the free run.

• cites [37] Kohler, K. & Ercole, A. Can network science reveal structure in a complex healthcare system? A network analysis using data from emergency surgical services. BMJ Open 10, 34265 (2020). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Presentations at ED are seasonal, so ideally models have more than one complete annual cycle to learn patterns from.

“Because presentations at ED are seasonal, ideally models have more than one complete annual cycle to learn patterns from11;”

• cites [11] Jones, R. Volatility in bed occupancy for emergency admissions. Br. J. Healthc. Manag. 17, 424–430 (2011). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

This study created complex models with many features, without dropping visits with missing data as other scholars have done, and included real-time location features that proved to be important for the long-staying patients who are more difficult to predict for.

“This study created complex models with many features, without dropping visits with missing data as other scholars have done20, and included real-time location features that proved to be important for the long-staying patients who are more difficult to predict for.”

• cites [20] El-Bouri, R., Eyre, D. W., Watkinson, P., Zhu, T. & Clifton, D. A. Hospital admission location prediction via deep interpretable networks for the year-round improvement of emergency patient care. IEEE J. Biomed. Health Inform. 25, 289–300 (2021). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

More heavily regularised LR models also performed well, suggesting that sparser models could potentially achieve acceptable performance in data-poor settings.

“More heavily regularised LR models also performed well, suggesting that sparser models could potentially achieve acceptable performance in data-poor settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The modular nature of the aggregation made it easy to swap out the bit that changed and retrain.

“Here, the modular nature of the aggregation made it easy to swap out the bit that changed and retrain.”

Classified as probably not a substantial claim. May combine two claims.

Ten-fold cross-validation was used to find hyperparameters for each model, optimising for log loss, and the validation set used to assess held-out performance of individual-level models and the whole pipeline during development.

“Ten-fold cross-validation was used to find hyperparameters for each model, optimising for log loss, and the validation set used to assess held-out performance of individual-level models and the whole pipeline during development41.”

• cites [41] Wilk, M. B. & Gnanadesikan, R. Probability plotting methods for the analysis for the analysis of data. Biometrika 55, 1–17 (1968).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

The challenge for predictions using ED data is the sparse and heterogenous nature of the recorded data.

“The challenge for predictions using ED data is the sparse and heterogenous nature of the recorded data.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The need for a post-hoc adjustment to models reflects wider difficulties with drift for all predictive applications that learn from past data, including but not limited to those using ML.

“The need for a post-hoc adjustment to models is reflective of wider difficulties with drift for all predictive applications that learn from past data, including but not limited to those using ML.”

May combine two claims.

The temporal split (test set most recent 20% of days, validation set the 10% before that) avoids any leak of future information into the past, and allows for a fairer test for problems with temporal drift.

“Visits were added to training, validation and test sets chronologically with the test set containing the most recent 20% of days, and the validation set the 10% before that, as shown in Supplementary Fig. 3. This temporal split avoids any leak of future information into the past, and allows for a fairer test for problems with temporal drift that are commonly in real-time implementation23.”

• cites [23] Nestor, B. et al. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Preprint at https://arxiv.org/abs/1908.00690 (2019).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

AI does not do anything on its own; to succeed, it must be connected to real-world processes.

“AI does not do anything on its own; to succeed, it must be connected to real-world processes38.”

• cites [38] Coiera, E. The last mile: where artificial intelligence meets reality. J. Med. Internet Res. 21, e16323 (2019). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The XGBoost and other classifiers used here have the benefit of being more interpretable than other algorithms such as neural networks, giving some insight into how the models derive their individual-level predictions.

“In addition to good predictive performance, the XGBoost and other classifiers used here have the benefit of being more interpretable than other algorithms such as neural networks, giving some insight into how the models derive their individual-level predictions.”

May combine two claims.

Peck et al.'s proposed summation of probabilities to calculate running bed need does not quantify uncertainty, and England et al.'s simulations do not benefit from real-time data.

“Peck et al.’s27 proposed summation of probabilities to calculate running bed need does not quantify uncertainty, and England et al.’s32 simulations do not benefit from real-time data.”

• cites [27] Peck, J. S., Benneyan, J. C., Nightingale, D. J. & Gaehde, S. A. Predicting emergency department inpatient admissions to improve same-day patient flow. Acad. Emerg. Med 19, E1045–E1054 (2012). sourceNot checked yet

  Not checked in the free run.

• cites [32] England, T. et al. Near real-time bed modelling feasibility study. J. Simul. 0, 1–12 (2019).Not checked yet

  Not checked in the free run.

May combine two claims.

The prediction pipeline generated output that could be evaluated at each step, which means that, when applied to data from later time periods, it is possible to identify (and potentially address) any weak step(s) in the pipeline.

“The prediction pipeline generated output that could be evaluated at each step, which means that, when applied to data from later time periods, it is possible to identify (and potentially address), any weak step(s) in the pipeline.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the real-time information product currently in use, a training set starting in November 2020 was used to train the models, and the survival curves are updated nightly to use only time-to-admission data from the last six weeks.

“So, for the real-time information product currently in use, a training set starting in November 2020 was used to train the models, and the survival curves are updated nightly to use only time-to-admission data from the last six weeks.”

• AssumptionRests on an assumption: Same Day Emergency Care (SDEC) was introduced late in 2020 and significantly changed how the ED is run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Changing patterns of ED presentations and/or changing operational practice may introduce drift or sudden changes in model performance, necessitating continuous monitoring.

“changing patterns of ED presentations and/or changing operational practice may introduce drift or sudden changes in model performance, necessitating continuous monitoring.”

May combine two claims.

To exploit the evolving predictive power of features, including data-completeness during patients' visits to ED, 12 temporally framed models were used; Supplementary Note 7 shows the value added by the use of 12 models compared to 3, confirming that additional models contribute to superior performance.

“To exploit the evolving predictive power of features, including data-completeness during patients’ visits to ED, 12 temporally framed models were used; Supplementary Note 7 shows the value added by the use of 12 models compared to 3 (the number chosen by Barak-Corren et al.31), confirming that additional models contribute to superior performance.”

• cites [31] Barak-Corren, Y. et al. Prediction across healthcare settings: a case study in predicting emergency department disposition. Npj Digit. Med. 4, 1–7 (2021). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Methods have been proposed for dealing with non-stationary learning problems.

“Methods have been proposed for dealing with non-stationary learning problems33,34.”

• cites [33] Wang, H. & Abraham, Z. Concept drift detection for streaming data. Preprint at http://arxiv.org/abs/1504.01044 (2015).Not checked yet

  Not checked in the free run.

• cites [34] Žliobaitė, I. Learning under concept drift: an overview. Preprint at http://arxiv.org/abs/1010.4784 (2010).Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Log loss was considered the most important metric, because, for input into the aggregation steps, accurate probabilities are more important than classification.

“Log loss was considered the most important metric, because, for input into the aggregation steps, accurate probabilities are more important than classification.”

Classified as probably not a substantial claim. May combine two claims.

XGBoost was chosen due to its efficiency and capability of processing large datasets, its ability to handle missing data and imbalanced datasets, and (relative to other ML algorithms) its interpretability.

“XGBoost was chosen due to its efficiency and capability of processing large datasets, its ability to handle missing data and imbalanced datasets, and (relative to other ML algorithms) its interpretability.”

• cites [40] Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. https://doi.org/10.1145/2939672.2939785 (2016). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A direct comparison with the six-week rolling average is only possible for a prediction window that ends at midnight.

“A direct comparison with it is only possible for a prediction window that ends at midnight.”