CoChat
Sign Up
Cited by 50Machine Learning in HealthcareMedical Coding and Health InformationArtificial Intelligence in Healthcare

Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes

Open access
Liu, Jinghui, Capurro, Daniel, Nguyen, Anthony, Verspoor, Karin·2021·npj Digital Medicine
DOI 10.1038/s41746-021-00474-9PMC PMC8249417
Open this paper in CoChat — free accountOpen PDF

Abstract

As healthcare providers receive fixed amounts of reimbursement for given services under DRG (Diagnosis-Related Groups) payment, DRG codes are valuable for cost monitoring and resource allocation. However, coding is typically performed retrospectively post-discharge. We seek to predict DRGs and DRG-based case mix index (CMI) at early inpatient admission using routine clinical text to estimate hospital cost in an acute setting. We examined a deep learning-based natural language processing (NLP) model to automatically predict per-episode DRGs and corresponding cost-reflecting weights on two cohorts (paid under Medicare Severity (MS) DRG or All Patient Refined (APR) DRG), without human coding efforts. It achieved macro-averaged area under the receiver operating characteristic curve (AUC) scores of 0·871 (SD 0·011) on MS-DRG and 0·884 (0·003) on APR-DRG in fivefold cross-validation experiments on the first day of ICU admission. When extended to simulated patient populations to estimate average cost-reflecting weights, the model increased its accuracy over time and obtained absolute CMI error of 2·40 (1·07%) and 12·79% (2·31%), respectively on the first day. As the model could adapt to variations in admission time, cohort size, and requires no extra manual coding efforts, it shows potential to help estimating costs for active patients to support better operational decision-making in hospitals.

AI snapshot

Written from the paper’s full text.

TL;DR

A deep learning NLP model using routine clinical notes predicted DRG groups and DRG-based case mix index early in ICU admission on MIMIC-III. It achieved strong population-level cost estimation, though individual DRG prediction remained limited.

Key findings

  • On the first day of ICU admission, the CAML NLP model achieved macro-AUC 0.871 (SD 0.011) for MS-DRG and 0.884 (0.003) for APR-DRG in fivefold cross-validation.
  • It outperformed an LSTM using 104 structured clinical measurements, which achieved macro-AUC 0.819 (0.009) for MS-DRG and 0.838 (0.017) for APR-DRG; the text model’s advantage was significant at p < 0.01 under a two-sided t-test.
  • Population CMI absolute error at 24 h post-admission was 2.40 (1.07%) for MS-DRG and 12.79% (2.31%) for APR-DRG; at 24 HPA, CMI error stayed under 8.0% for MS-DRG and 15.0% for APR-DRG across cohort sizes from 200 to over 1500.
  • Individual DRG F1 scores were low: micro-F1 was 0.270 (0.006) for MS-DRG and 0.244 (0.005) for APR-DRG on all cases, improving to 0.329 (0.008) and 0.306 (0.004) for DRGs covering 80% of test stays.

Methods

  • Retrospective study using MIMIC-III ICU data, creating MS-DRG and APR-DRG cohorts from hospital stays with patients over 18 and one ICU visit; clinical notes charted up to 48 h after ICU admission were included, with train/test splits by patient and fivefold cross-validation on 90% of data after a 10% hold-out test set.
  • Clinical notes were sorted by chart time, concatenated, tokenized, lowercased, mapped to pretrained biomedical/clinical word embeddings, and truncated or padded to 2000 tokens; the main model was CAML, a CNN with attention adapted for single-label DRG prediction, compared against an LSTM trained on 104 structured clinical measurements.
  • Evaluation used macro- and micro-averaged AUC and F1 for DRG prediction, and absolute CMI error for simulated patient populations across varying hours post-admission and population sizes; payment estimation assumed a base payment rate of $6000.

Limitations

  • Stated: the study used only one US medical center (MIMIC-III), limiting location and patient representativeness; de-identification prevented exact fiscal-year DRG weight mapping, so official FY2013 weights were used; the focus on ICU patients leaves generalizability to other inpatients uncertain.
  • Stated: individual-level DRG prediction remains inadequate for decision support, with challenges in extracting diagnostic evidence, identifying major versus negligible comorbidities, handling rare or unseen DRGs, and modeling dynamic patient trajectories; CC/MCC differentiation and different clinical text functions were difficult.
  • Evident from the text: class imbalance inflates AUC while F1 scores remain low; APR-DRG had more prediction targets and generally higher CMI errors than MS-DRG; the model was not prospectively validated, and NLP methodology was not comprehensively examined.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-021-00474-9

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 29 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    At 24 HPA, the model contained CMI error under 8.0% for MS-DRG and 15.0% for APR-DRG with cohort sizes ranging from 200 to over 1500.

    “Figure 2 demonstrates that at 24 HPA the model contained the CMI error under 8·0% for MS-DRG and 15·0% for APR-DRG with different cohort sizes, ranging from 200 to over 1500.”
    • Also stated in: § Discussion · hedged
      “Our results showed the machine learning-based prediction could contain the CMI error under 8·0% for MS-DRG and 15·0% for APR-DRG on the first day of ICU admission.”
    Quote in the paperSubstantial claim
  • Findingown data

    CMI prediction errors tended to stabilize by 48 HPA.

    “The predicted CMI made by the NLP model on these two sets achieved errors of 2·40 (1·07%) and 12·79% (2·31%) on 24 h post-admission (HPA), or the first day of ICU, and tended to stabilize by 48 HPA (detailed numbers in Supplementary Table 2).”
    Quote in the paperSubstantial claim
  • Findingown data

    When applied to predict DRG payment weights in regression, the model achieved mean absolute error (MAE) under 1.51 for both DRG cohorts.

    “We presented the approach in Supplementary Method 3 to apply the model to predict DRG payment weights in regression, where the model achieved mean absolute error (MAE) under 1·51 for both DRG cohorts (see Supplementary Table 1).”
    Quote in the paperNot a substantial claim
  • Findingown data

    The NLP model achieved macro-averaged AUC scores of 0.871 (SD 0.011) on MS-DRG and 0.884 (0.003) on APR-DRG in fivefold cross-validation experiments on the first day of ICU admission.

    “It achieved macro-averaged area under the receiver operating characteristic curve (AUC) scores of 0·871 (SD 0·011) on MS-DRG and 0·884 (0·003) on APR-DRG in fivefold cross-validation experiments on the first day of ICU admission.”
    • Also stated in: § DRG prediction · plain
      “For both MS- and APR-DRG, the NLP model achieved macro-averaged area under the receiver operating characteristic curve (AUC) over 0·86 and micro-averaged AUC over 0·95.”
    Quote in the paperSubstantial claim
  • Findingown data

    For common DRG codes accounting for 80% of total test stays, the model's micro-F1 improved to 0.329 (0.008) on MS-DRG and 0.306 (0.004) on APR-DRG.

    “Meanwhile, when looking at common DRG codes that account of 80% of total test stays, the results improved to 0·329 (0·008) and 0·306 (0·004), respectively and would further boost when examining more frequent DRGs.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 29
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 23

  • May combine two claims.

    Findingown data

    The model tended to underestimate population payments in the beginning and gradually became more reliable, with MS-DRG approaching true payment amount and APR-DRG stabilized at slightly lower level.

    “Figure 3 shows the model tended to underestimate population payments in the beginning and gradually became more reliable, with MS-DRG approaching true payment amount and APR-DRG stabilized at slightly lower level.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: A base payment rate of $6000 for the hospital.

  • May combine two claims.

    Findingown data

    The LSTM reference model trained on time-series constructed on 104 clinical measurements achieved macro-AUC of 0.819 (0.009) for MS-DRG and 0.838 (0.017) for APR-DRG on all DRG codes.

    “Trained on time-series constructed on 104 clinical measurements using long short-term memory (LSTM)13, this model achieved macro-AUC of 0·819 (0·009) and 0·838 (0·017) for MS-DRG APR-DRG on all DRG codes.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997). source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The NLP model achieved absolute CMI error of 2.40 (1.07%) for MS-DRG and 12.79% (2.31%) for APR-DRG at 24 h post-admission (first day of ICU).

    “When extended to simulated patient populations to estimate average cost-reflecting weights, the model increased its accuracy over time and obtained absolute CMI error of 2·40 (1·07%) and 12·79% (2·31%), respectively on the first day.”
    • Also stated in: § CMI prediction · plain
      “The predicted CMI made by the NLP model on these two sets achieved errors of 2·40 (1·07%) and 12·79% (2·31%) on 24 h post-admission (HPA), or the first day of ICU, and tended to stabilize by 48 HPA (detailed numbers in Supplementary Table 2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model achieved macro-AUCs of 0.836 (0.016), 0.850 (0.011), and 0.833 (0.013) on MS-DRG for diseases and disorders of the circulatory system, nervous system, and respiratory system, respectively, and 0.881 (0.003), 0.892 (0.004), and 0.759 (0.012) for APR-DRG.

    “The model was able to achieve macro-AUCs (SD) of 0·836 (0·016), 0·850 (0·011), and 0·833 (0·013) on MS-DRG for diseases and disorders of the circulatory system, nervous system, and respiratory system, respectively, and similarly 0·881 (0·003), 0·892 (0·004), and 0·759 (0·012) for APR-DRG.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The NLP-based model consistently performed superior to the model using structured clinical data (p < 0.01 under two-sided t-test), with strongest contrast on F1 scores.

    “Although strong performances were observed with structured data, the NLP-based model consistently performed superior than the model using structured clinical data (p value < 0.01 under two-sided t-test), with strongest contrast on F1 scores.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CMI prediction significantly improved around ICU admission (−6 HPA and 6 HPA) and error reached lower than 5.0% for MS-DRG and 15.0% for APR-DRG after 24 HPA.

    “The CMI prediction significantly improved around the ICU admission (−6 HPA and 6 HPA) and error reached lower than 5·0% for MS-DRG and 15·0% for APR-DRG after 24 HPA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model still achieved a mean CMI error below 15.0% for APR-DRG and was more robust against changes in cohort size.

    “Nevertheless, the model still achieved a mean below 15·0% and was more robust against changes in cohort size.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    The model obtained micro-F1 scores of 0.270 (0.006) on MS-DRG and 0.244 (0.005) on APR-DRG on all cases.

    “The performance was also evaluated using F1 scores, where the model obtained 0·270 (0·006) and 0·244 (0·005) of micro-F1 on all cases in the two cohorts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CAML was found to be a simple yet effective baseline compared against large pretrained models like BERT and ClinicalBERT, which provided improved AUC but lower F1 scores compared to CAML in both cohorts.

    “Meanwhile, we found CAML to be a simple yet effective baseline even compared against large pretrained models like BERT26 and ClinicalBERT27 (the domain adapted version of BERT), which provided improved AUC but lower F1 scores compared to CAML in both cohorts.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [26] Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Burstein, J., Doran, C. & Solorio, T) 4171–4186 (Association for Computational Linguistics, 2019).

      Not checked in the free run.

    • cites [27] Alsentzer, E. et al. Publicly available clinical BERT embeddings. In Proc. 2nd Clinical Natural Language Processing Workshop (eds Rumshisky, A., Roberts, K., Bethard, S. & Naumann, T.) 72–78 (Association for Computational Linguistics, 2019). source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    For both MS- and APR-DRG, the NLP model achieved macro-averaged AUC over 0.86 and micro-averaged AUC over 0.95.

    “For both MS- and APR-DRG, the NLP model achieved macro-averaged area under the receiver operating characteristic curve (AUC) over 0·86 and micro-averaged AUC over 0·95.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Error analysis showed the model correctly predicted many true negatives for each DRG given the class imbalance, thus increasing the average AUC performance, whereas the F1 scores reflect errors on false positive and false negative predictions.

    “Error analysis showed the model correctly predicted many true negatives for each DRG given the class imbalance, thus increasing the average AUC performance, whereas the F1 scores reflect the errors on false positive and false negative predictions requiring future improvement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Case 4, the model predicted the more frequent DRG 023, while the true DRG was 955.

    “such as the model predicted the more frequent DRG 023 for Case 4, whose true DRG is in fact 955 (both DRGs are related to craniotomy).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Case 2, the model correctly extracted text on pneumonia and classified the case as DRG 193.

    “For example, in Case 2, the model correctly extracted text on pneumonia, thus classifying the case as DRG 193, but failed to recognize its diagnostic relation to more complicated conditions like sepsis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DRG 289 appeared only once in the whole cohort and was in the test set, so the model had no training sample of that DRG.

    “Case 5 is an extreme case where DRG 289 only appeared once in the whole cohort, and it was in the test set, creating a zero-shot learning scenario for the model as it had not seen any training sample of the DRG.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    By forecasting the overall CMI of an in-hospital population, the model prediction may help administrators foresee peaks and troughs in resource usage and better arrange resources such as staffing and operating theaters.

    “By forecasting the overall CMI of an in-hospital population, the model prediction may help administrators foresee peaks and troughs in resource usage and better arrange resources such as staffing and operating theaters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    DRG prediction using clinical text consistently performed better than clinical measurements alone, demonstrating the value of mining clinical text to support active care management.

    “DRG prediction using clinical text consistently performed better than clinical measurements alone, demonstrating the value of mining clinical text to support active care management.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Studies have shown the promise of machine learning in predicting events like unexpected 30-day readmission and identifying high-need high-cost patients that could preempt avoidable health expenses.

    “Also, studies have shown the promise of machine learning in predicting events like unexpected 30-day readmission8,23 and identifying high-need high-cost patients24 that could preempt avoidable health expenses.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [8] Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. 1, 18 (2018). source

      Not checked in the free run.

    • cites [23] Frizzell, J. D. et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol. 2, 204–209 (2017). source

      Not checked in the free run.

    • cites [24] Osawa, I., Goto, T., Yamamoto, Y. & Tsugawa, Y. Machine-learning-based prediction models for high-need high-cost patients using nationwide clinical and claims data. npj Digit. Med. 3, 148 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing studies examined health cost prediction from a payer perspective, such as using claims data to predict cost in subsequent years.

    “Existing studies21,22 examined health cost prediction from a payer perspective, such as using claims data to predict cost in subsequent years.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [21] Morid, M. A., Kawamoto, K., Ault, T., Dorius, J. & Abdelrahman, S. Supervised learning methods for predicting healthcare costs: systematic literature review and empirical evaluation. AMIA Annu. Symp . Proc. 2017, 1312–1321 (2017).

      Not checked in the free run.

    • cites [22] Morid, M. A., Sheng, O. R. L., Kawamoto, K. & Abdelrahman, S. Learning hidden patterns from patient multivariate time series data using convolutional neural networks: a case study of healthcare cost prediction. J. Biomed. Inform. 111, 103565 (2020). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The prior study utilized derived patient-related data, including coded diagnosis and approximated DRG results, and evaluated the model with hospital-specific information on resources.

    “However, the study utilized derived patient-related data, including coded diagnosis and approximated DRG results, and evaluated the model with hospital-specific information on resources.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] Gartner, D., Kolisch, R., Neill, D. B. & Padman, R. Machine learning approaches for early DRG classification and resource allocation. INFORMS J. Comput. 27, 718–734 (2015). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    A prior study applied statistical machine learning for early DRG prediction and resource allocation at a 350-bed hospital in Germany, examined a range of feature selection and classification techniques in different prediction settings, and showed their DRG predictions were relevant for optimizing scarce hospital resources allocation and the improvement of contribution margin for the hospital.

    “A prior study7 applied statistical machine learning for early DRG prediction and resource allocation at a 350-bed hospital in Germany, examined a range of feature selection and classification techniques in different prediction settings, and showed their DRG predictions were relevant for optimizing scarce hospital resources allocation and the improvement of contribution margin for the hospital.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] Gartner, D., Kolisch, R., Neill, D. B. & Padman, R. Machine learning approaches for early DRG classification and resource allocation. INFORMS J. Comput. 27, 718–734 (2015). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The better MS-DRG performance could be due to APR-DRG having more fine-grained severity stratification and hence more prediction targets (1136 vs. 738).

    “Finally, performances in MS-DRG were in general better than APR-DRG, which could be due to APR-DRG having more fine-grained severity stratification and hence more prediction targets (1136 vs. 738).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: APR-DRG has more fine-grained severity stratification and hence more prediction targets (1136 vs. 738) than MS-DRG.

  • May combine two claims.

    Interpretationown data

    The deep learning-based NLP model could extract indicators to infer a reasonable DRG group for a patient stay, especially for diseases of the circulatory system (MDC 05) and frequent DRGs, but accurately predicting each individual DRG remains challenging.

    “Though the deep learning-based NLP model could extract indicators to infer a reasonable DRG group for a patient stay, especially for diseases of circulatory system (MDC 05) and frequent DRGs, accurately predicting each individual DRG is still a challenging task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    When calculating CMI, mistakes on individual cases offset each other to drive down the overall error, and the performances were robust when the patient population size changed.

    “When calculating CMI, mistakes on individual cases offset each other to drive down the overall error, and we showed such performances were robust when the patient population size changed.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    The current examination of MS-DRG and APR-DRG showed the flexibility of the modeling approach on different DRG systems, which may be extended to other DRGs with similar structures of grouping and weight.

    “The current examination of MS-DRG and APR-DRG showed the flexibility of the modeling approach on different DRG systems, which may be extended to other DRGs with similar structures of grouping and weight.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Precise cost information based on well-represented patient data may assist payers in adjusting payment policies to further increase efficiency and support new payment initiatives.

    “At the same time, precise cost information based on well-represented patient data may also assist payers in adjusting payment policies to further increase efficiency and support new payment initiatives19,20.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [19] Press, M. J., Rajkumar, R. & Conway, P. H. Medicare’s new bundled payments: design, strategy, and evolution. J. Am. Med. Assoc. 315, 131–132 (2016). source

      Not checked in the free run.

    • cites [20] Malik, A. T., Phillips, F. M., Yu, E. & Khan, S. N. Are current DRG-based bundled payment models for lumbar fusions risk-adjusting adequately? An analysis of medicare beneficiaries. Spine J. 20, 32–40 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Early DRG prediction involves the challenge of extracting diagnostic evidence.

    “Early DRG prediction involves several challenges, including extracting diagnostic evidence, identifying major or negligible comorbidities, handling rare or unseen test DRGs, and modeling dynamic patient trajectories.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Deep neural network models have the potential to identify important diagnostic indicators from raw data and encapsulate clinical patterns.

    “By leveraging a large set of patient data, deep neural network models have the potential to identify important diagnostic indicators from the raw data and encapsulate clinical patterns8–10.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [8] Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. npj Digit. Med. 1, 18 (2018). source

      Not checked in the free run.

    • cites [9] Rajkomar, A., Dean, J. & Kohane, I. Machine learning in medicine. N. Engl. J. Med. 380, 1347–1358 (2019). source

      Not checked in the free run.

    • cites [10] Xiao, C., Choi, E. & Sun, J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J. Am. Med. Inform. Assoc. 25, 1419–1428 (2018). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Healthcare expenditure has become a significant component of global spending, and many health systems face challenges to improve efficiency and provide sustainable care.

    “Healthcare expenditure has become a significant component of global spending14, and many health systems face challenges to improve efficiency and provide sustainable care15,16.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [14] World Health Organization (WHO). Global Spending on Health: A World in Transition 2019 (WHO, 2019).

      Not checked in the free run.

    • cites [15] Chang, A. Y. et al. Past, present, and future of global health financing: a review of development assistance, government, out-of-pocket, and other private spending on health for 195 countries, 1995-2050. Lancet 393, 2233–2260 (2019). source

      Not checked in the free run.

    • cites [16] Dieleman, J. L. et al. US Health Care spending by payer and health condition, 1996-2016. J. Am. Med. Assoc. 323, 863–884 (2020). source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The datasets were split by patient instead of by admission for both test set and cross-validation, ensuring that a patient would not appear in both cross-validation and testing and therefore avoiding data leakage.

    “The datasets were split by patient instead of by admission for both test set and cross-validation, ensuring that a patient would not appear in both cross-validation and testing and therefore avoiding data leakage.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Although the study achieved favorable results on population CMI, the modeling method of clinical notes remains to be improved to make more accurate DRG predictions at an individual level.

    “Thirdly, though achieving favorable results on population CMI, the modeling method of clinical notes remains to be improved to make more accurate DRG predictions at an individual level.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The DRG groups and their corresponding weights are pre-defined, so a provider can expect reimbursement for an assigned DRG by multiplying the relative weight by a fixed dollar amount (the base payment rate).

  • May combine two claims.

    Interpretationown data

    The machine learning model can suffer from a lack of enough training data and make errors on rare DRGs.

    “In addition, the machine learning model could also suffer from a lack of enough training data and make errors on rare DRGs, such as the model predicted the more frequent DRG 023 for Case 4, whose true DRG is in fact 955 (both DRGs are related to craniotomy).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study kept 10% of data as a hold-out test set for each DRG system and performed fivefold cross-validation on the remaining 90%.

    “We first kept 10% of data as the hold-out test set for each DRG system and then performed fivefold cross-validation, training, and tuning the hyper-parameters of independent models on five different splits of the remaining 90% of data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Early DRG prediction involves the challenge of modeling dynamic patient trajectories.

    “Early DRG prediction involves several challenges, including extracting diagnostic evidence, identifying major or negligible comorbidities, handling rare or unseen test DRGs, and modeling dynamic patient trajectories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hospital operation managers may review quarterly or even monthly DRG-based statistics to assess patient mix and financial efficiency under DRG reimbursement.

    “Hospital operation managers may review quarterly or even monthly DRG-based statistics to assess its patient mix3,4 and financial efficiency5,6 under DRG reimbursement.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3] Mendez, C. M., Harrington, D. W., Christenson, P. & Spellberg, B. Impact of hospital variables on case mix index as a marker of disease severity. Popul. Health Manag. 17, 28–34 (2014). source

      Not checked in the free run.

    • cites [4] Thompson, N. D., Edwards, J. R., Dudeck, M. A., Fridkin, S. K. & Magill, S. S. Evaluating the use of the case mix index for risk adjustment of healthcare-associated infection data: an illustration using clostridium Difficile infection data from the national healthcare safety network. Infect. Control Hosp. Epidemiol. 37, 19–25 (2016). source

      Not checked in the free run.

    • cites [5] Quinn, K. After the revolution: DRGs at age 30. Ann. Intern. Med. 160, 426–429 (2014).

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The case mix index (CMI) is an averaged score of DRG weights given a patient group.

    “we adopted the notion of the CMI3,25, which is an averaged score of DRG weights given a patient group.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [3] Mendez, C. M., Harrington, D. W., Christenson, P. & Spellberg, B. Impact of hospital variables on case mix index as a marker of disease severity. Popul. Health Manag. 17, 28–34 (2014). source

      Not checked in the free run.

    • cites [25] Kuster, S. P. et al. Correlation between case mix index and antibiotic use in hospitals. J. Antimicrob. Chemother. 62, 837–842 (2008). source

      Not checked in the free run.

    • Rests on an assumption: The DRG groups and their corresponding weights are pre-defined, so a provider can expect reimbursement for an assigned DRG by multiplying the relative weight by a fixed dollar amount (the base payment rate).

  • May combine two claims.

    Interpretationown data

    CC and MCC play significant roles in DRG payment but are challenging to differentiate based on text.

    “As shown in the case analyses, CC and MCC play significant roles in the DRG payment but are challenging to differentiate based on text.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The DRG groups and their corresponding weights are pre-defined, so a provider can expect reimbursement for an assigned DRG by multiplying the relative weight by a fixed dollar amount (the base payment rate).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    When evaluating cost estimation on simulated populations, the test set was trimmed by keeping only one hospital admission for each unique patient.

    “When evaluating cost estimation on simulated populations, the test set was further trimmed by keeping only one hospital admission for each unique patient to form the test cohort.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Early DRG prediction involves the challenge of identifying major or negligible comorbidities.

    “Early DRG prediction involves several challenges, including extracting diagnostic evidence, identifying major or negligible comorbidities, handling rare or unseen test DRGs, and modeling dynamic patient trajectories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One-dimensional CNN can be understood as extracting predictive n-gram features and encoding these features in a latent representation.

    “When modeling sequential data, one-dimensional CNN can be understood intuitively as extracting predictive n-gram features and encoding these features in a latent representation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Kim, Y. Convolutional neural networks for sentence classification. In Proc. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (eds Moschitti, A., Pang, B. & Daelemans, W.) 1746–1751 (Association for Computational Linguistics, 2014). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The CAML architecture was shown to be effective for automatic ICD coding.

    “In this study, we adopted an architecture shown to be effective for automatic ICD coding named CAML12 that pools CNN features using the attention mechanism, modifying it for the single-label DRG prediction task.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12] Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J. & Eisenstein, J. Explainable prediction of medical codes from clinical text. In Proc. of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Walker, M., Ji, H. & Stent, A.) 1101–1111 (Association for Computational Linguistics, 2018). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Various payment incentives have been proposed and experimented with curbing the rising cost in the healthcare system to encourage healthcare providers to plan discreetly on resources, promoting risk-sharing between payers and providers.

    “Various payment incentives have been proposed and experimented with curbing the rising cost in the healthcare system to encourage healthcare providers to plan discreetly on resources17,18, promoting risk-sharing between payers and providers.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [17] Baker, L. C., Bundorf, M. K., Devlin, A. M. & Kessler, D. P. Medicare advantage plans pay hospitals less than traditional medicare pays. Health Aff. 35, 1444–1451 (2016). source

      Not checked in the free run.

    • cites [18] James, B. C. & Poulsen, G. P. The case for capitation. Harv. Bus. Rev. 94, 102–111 (2016).

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisereasoning

    These prior studies did not focus on active patients to provide support for provider decisions.

    “Meanwhile, these studies did not focus on active patients to provide support for provider decisions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Early DRG prediction involves the challenge of handling rare or unseen test DRGs.

    “Early DRG prediction involves several challenges, including extracting diagnostic evidence, identifying major or negligible comorbidities, handling rare or unseen test DRGs, and modeling dynamic patient trajectories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The DRG payment system has become a significant component of healthcare payments in many countries to promote risk-sharing between healthcare providers and payers.

    “The payment system based on diagnosis-related groups, or DRGs, was designed to manage healthcare costs and maintain sustainable operations for inpatients and it has become a significant component of healthcare payments in many countries to promote risk-sharing between healthcare providers and payers1,2.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] Bredenkamp, C., Bales, S. & Kahur, K. Transition to Diagnosis-Related Group (DRG) Payments for Health: Lessons from Case Studies (The World Bank, 2019). source

      Not checked in the free run.

    • cites [2] Mihailovic, N., Kocic, S. & Jakovljevic, M. Review of diagnosis-related group-based financing of hospital care. Heal. Serv. Res. Manag. Epidemiol. https://doi.org/10.1177/2333392816647892 (2016). source

      Not checked in the free run.

44 of 44

  • The paper relies on this without support.

    DRGs are typically obtained after patient discharge, making it impossible to act upon this information to make adjustments for active patients.

    “Meanwhile, such review is limited to retrospective assessment since DRGs are typically obtained after patient discharge, making it impossible to act upon this information to make adjustments for active patients.”

  • The paper relies on this without support.

    Calculating DRGs is a time-consuming process requiring expert efforts to manually identify information from patient records, standardize it to ICD format, and then obtain DRGs.

    “Furthermore, calculating DRGs is a time-consuming process requiring expert efforts to manually identify information from patient records, standardize it to ICD (International Classification of Diseases) format, and then obtain DRGs.”

  • The authors state this as an assumption.

    A base payment rate of $6000 for the hospital.

    “By assuming a base payment rate of $6000 and approximating DRG-based payment by the product of the rate and DRG weight, we could also estimate the DRG payment amount for a hospital.”
    “Assuming a base payment rate of $6000 for the hospital, the CMI error rate of ~5·0% may lead to ~$472,500 difference in final reimbursement for a simulated cohort of 500 patients with a CMI of 3·15, but it allows the hospital to learn about its CMI at an early stage using only routine data and make decisions accordingly, implementing resource allocation and revenue capture.”

  • The paper relies on this without support.

    Population CMI is adopted by hospitals and payers as a cost indicator to review clinical complexity and efficiency.

    “Population CMI, which averages over a set of patient DRG weights, is adopted by hospitals and payers as a cost indicator to review clinical complexity and efficiency.”

  • The paper relies on this without support.

    APR-DRG has more fine-grained severity stratification and hence more prediction targets (1136 vs. 738) than MS-DRG.

    “Finally, performances in MS-DRG were in general better than APR-DRG, which could be due to APR-DRG having more fine-grained severity stratification and hence more prediction targets (1136 vs. 738).”

  • The paper relies on this without support.

    The rapid spread of electronic health records has created large amounts of patient data and provides an opportunity to estimate DRGs and related costs at early patient admission using machine learning.

    “The rapid spread of electronic health records (EHRs) has created large amounts of patient data and provides an opportunity to estimate DRGs and related costs at early patient admission using machine learning.”

  • The paper relies on this without support.

    An inpatient hospital admission is expected to receive a single DRG code under a specific DRG payment system to process claims with a payer.

    “Under the DRG payment system, an inpatient hospital admission is expected to receive a single DRG code under a specific DRG payment system to process claims with a payer.”

  • The paper relies on this without support.

    The DRG groups and their corresponding weights are pre-defined, so a provider can expect reimbursement for an assigned DRG by multiplying the relative weight by a fixed dollar amount (the base payment rate).

    “During payment, the DRG groups and their corresponding weights are pre-defined, so a provider can expect reimbursement for an assigned DRG by multiplying the relative weight by a fixed dollar amount, referred to as the base payment rate, which is specific to the provider based on local factors like wages.”

  • The paper relies on this without support.

    Intensive care for ICU patients usually generates more data than other inpatients, providing more signals for a learning model.

    “On the other hand, intensive care for ICU patients usually generates more data than other inpatients, providing more signals for a learning model.”

  • The paper relies on this without support.

    The acute conditions of ICU patients are more dynamic and less predictable.

    “On the one hand, the acute conditions of ICU patients are more dynamic and less predictable, and strong performances on them may indicate the capacity of the approach to model other inpatients, which should be studied.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MS-DRG top-80% row does not report a stay count

    Minor

    The final column is defined as the number and percentage of hospital stays covered. For the MS-DRG top-80% subset it instead repeats the macro-AUC value and SD, so it cannot be the stated stay count and percentage.

1 of 1

Cites

Flagged first, then by citations

2 of the 33 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Andrew, S., O’Reilly, J., Ward, P. & Mason, A. in Diagnosis-related Groups in Europe: Moving towards Transparency, Efficiency and Quality in Hospitals (eds Busse, R., Geissler, A., Quentin, W. & Wiley, M.) Ch. 7 (McGraw-Hill Education, 2011).
  2. Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Burstein, J., Doran, C. & Solorio, T) 4171–4186 (Association for Computational Linguistics, 2019).
  3. Jain, S. & Wallace, B. C. Attention is not explanation. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Burstein, J., Doran, C. & Solorio, T) 3543–3556 (Association for Computational Linguistics, 2019).

Cited by

Sorted by citations

Related

Authors
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K.
Affiliations
  • Commonwealth Scientific and Industrial Research Organisation (AU)
  • The University of Melbourne (AU)
Corresponding author
Verspoor, Karin
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2021-07-01
Volume
4
Issue
1
Pages
103-103
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-021-00474-9
PMC
PMC8249417
OpenAlex
W3174700141
Semantic Scholar
93537c66ffb6977649444a5325ae56965a2266cf
Cited by
50
References
33
Citations by year
2026: 5 · 2025: 13 · 2024: 11 · 2023: 6 · 2022: 14 · 2021: 1
FWCI
2.83
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
  • RMIT Research Repository (RMIT University Library) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
  • Department of Health | National Health and Medical Research Council (1134919)
  • Commonwealth Scientific and Industrial Research Organisation
  • University of Melbourne, Melbourne School of Engineering
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
diagnosis-related groups, hospital cost estimation, clinical notes, natural language processing, case mix index, intensive care unit admission, hospital reimbursement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review