DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Open accessPublished version of a preprint
Abstract
In the U.S. inpatient payment system, the Diagnosis-Related Group (DRG) is pivotal, but its assignment process is inefficient. The study introduces , an advanced large language model (LLM) fine-tuned on clinical notes to enhance DRGs assignment. Utilizing LLaMA as the foundational model and optimizing it through Low-Rank Adaptation (LoRA) on 236,192 MIMIC-IV discharge summaries, our -7B model exhibited a noteworthy macro-averaged F1 score of 0.327, a top-1 prediction accuracy of 52.0%, and a macro-averaged Area Under the Curve (AUC) of 0.986, with a maximum input token length of 512. This model surpassed the performance of prior leading models in DRG prediction, showing a relative improvement of 40.3% and 35.7% in macro-averaged F1 score compared to ClinicalBERT and CAML, respectively. Applied to base DRG and complication or comorbidity (CC)/major complication or comorbidity (MCC) prediction, achieved a top-1 prediction accuracy of 67.8% and 67.5%, respectively. Additionally, our findings indicate that ’s performance correlates with increased model parameters and input context lengths.
AI snapshot
Written from the paper’s full text.
TL;DR
DRG-LLaMA fine-tunes LLaMA with LoRA on MIMIC-IV discharge summaries to predict diagnosis-related groups (DRGs) from clinical text. It outperforms ClinicalBERT and CAML on DRG prediction and shows improved performance with larger model sizes and longer input contexts.
Key findings
- With a 7B LLaMA model and max input token size 512, DRG-LLaMA achieved macro-F1 0.327, top-1 accuracy 52.0%, top-5 accuracy 84.8%, and macro-AUC 0.986, with relative macro-F1 improvements of 40.3% vs. ClinicalBERT and 35.7% vs. CAML.
- Performance improved with larger models and longer contexts: the 13B model with 1024 max input tokens reached top-1 accuracy 54.6%, top-5 accuracy 86.5%, and macro-F1 0.361.
- In the two-label approach, base DRG and CC/MCC top-1 accuracies were 67.8% and 67.5%, respectively; combining them via mapping rules yielded 51.5% final DRG accuracy, comparable to the 52.0% single-label result.
- Accuracy was higher for more frequent DRGs: top-1 accuracy was 55.7% for the top 300 DRGs and 69.4% for the top 30 DRGs, while performance declined for less frequent DRGs.
Methods
- Design: fine-tuned LLaMA using Low-Rank Adaptation (LoRA) for DRG prediction from clinical notes, testing both single-label DRG classification and two-label prediction of base DRG plus CC/MCC status.
- Data: MIMIC-IV discharge summaries, using the “brief hospital course” section; MS-DRG codes were consolidated to version 34.0, with 738 DRGs represented, a 90%/10% training/testing split stratified by DRG, and 26,244 test cases.
- Analysis: compared against CAML and ClinicalBERT; reported macro/micro F1, macro/micro AUC, top-1/5/10 accuracy, with standard deviations from 30 bootstrap iterations.
Limitations
- Stated: the study was limited to MIMIC-IV discharge summaries, which are available only after discharge, so early DRG prediction using admission or ED notes was not tested; computational limits restricted experiments to LLaMA models up to 13B parameters and prevented extensive hyperparameter search.
- Evident: performance declined for rare DRGs, and the paper notes noisy CC/MCC assignment and cases where correct DRG prediction required information not available in discharge summaries.
- Evident: manual error analysis was based on only 10 randomly selected erroneous cases, limiting generalizability of the error categories.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 38 cited sources checked · 0 supported · 0 flagged
- Findingown data
DRG-LLaMA-7B achieved a top-5 prediction accuracy of 84.8% on DRG prediction with a maximum input token size of 512.
“The accuracy of top-1 and top-5 predictions achieved by our fine-tuned DRG-LLaMA -7B model was 52.0% and 84.8%, respectively.”
Quote in the paperSubstantial claim - Findingreasoning
Out of the DRGs with a top-1 prediction accuracy of 100%, 8 out of 9 are surgical DRGs.
“out of the DRGs with a top-1 prediction accuracy of 100%, 8 out of 9 are surgical DRGs”
Quote in the paperNot a substantial claim - Findingown data
A correlation exists between the number of training cases and prediction performance.
“As noted above, a correlation exists between the number of training cases and prediction performance.”
Quote in the paperNot a substantial claim - Findingown data
DRG-LLaMA's performance consistently improved with larger models and longer input contexts.
“observing a consistent improvement in all evaluation metrics with larger models and longer input contexts”
- Also stated in: § DRG prediction as a single-label classification task · plain
“We observed that DRG-LLaMA ’s performance consistently improved with larger models and longer input contexts.”
Quote in the paperSubstantial claim - Also stated in: § DRG prediction as a single-label classification task · plain
- Findingown data
The optimal configuration of DRG-LLaMA, using a 13B LLaMA model and a maximum input token size of 1024, achieved a top-1 prediction accuracy of 54.6%.
“The optimal configuration, utilizing a 13B LLaMA model and a maximum input token size of 1024, achieved a top-1 prediction accuracy of 54.6%”
Quote in the paperSubstantial claim
- Findingown data
DRG-LLaMA-7B showed a relative improvement of 40.3% in macro-averaged F1 score compared to ClinicalBERT.
“showing a relative improvement of 40.3% and 35.7% in macro-averaged F1 score compared to ClinicalBERT and CAML, respectively”
Quote in the paperSubstantial claim - Findingown data
When considering only the most frequent 300 DRGs, DRG-LLaMA-7B's top-1 accuracy improved to 55.7%.
“When only considering the most frequent 300 DRGs, the top-1 accuracy improved to 55.7%”
Quote in the paperNot a substantial claim - Findingown data
When considering only the most frequent 30 DRGs, DRG-LLaMA-7B's top-1 accuracy improved to 69.4%.
“this further increased to 69.4% in the most frequent 30 DRGs”
Quote in the paperNot a substantial claim - Findingown data
DRGs with a top-5 prediction accuracy exceeding 80% are associated with a median of 309 training cases per label.
“DRGs with a top-5 prediction accuracy exceeding 80% are associated with a median of 309 training cases per label.”
Quote in the paperNot a substantial claim - Findingown data
The optimal configuration of DRG-LLaMA achieved a macro-F1 score of 0.361.
“The optimal configuration, utilizing a 13B LLaMA model and a maximum input token size of 1024, achieved a top-1 prediction accuracy of 54.6%, a top-5 prediction accuracy of 86.5%, and a macro-F1 score of 0.361.”
Quote in the paperSubstantial claim - Findingown data
DRGs with a top-5 accuracy below 20% are associated with a median of 17 training cases per label.
“those DRGs with a top-5 accuracy below 20% are associated with only a median of 17 training cases per label”
Quote in the paperNot a substantial claim - Findingown data
DRG-LLaMA-7B showed a relative improvement of 35.7% in macro-averaged F1 score compared to CAML.
“showing a relative improvement of 40.3% and 35.7% in macro-averaged F1 score compared to ClinicalBERT and CAML, respectively”
Quote in the paperSubstantial claim - Findingown data
Upon integrating a mapping rule to infer DRGs from base DRG and CC/MCC labels, accuracy reached 51.5% across all DRGs.
“Upon integrating a mapping rule designed to infer DRGs through the combination of base DRG and CC/MCC labels, the accuracy reached 51.5% across all DRGs.”
Quote in the paperSubstantial claim - Findingown data
In the two-label approach, DRG-LLaMA-7B achieved a top-1 prediction accuracy of 67.5% for CC/MCC.
“The top-1 prediction accuracy for base DRG and CC/MCC reached 67.8% and 67.5%, respectively.”
Quote in the paperSubstantial claim - Findingown data
In the two-label approach, DRG-LLaMA-7B achieved a top-1 prediction accuracy of 67.8% for base DRG.
“The top-1 prediction accuracy for base DRG and CC/MCC reached 67.8% and 67.5%, respectively.”
Quote in the paperSubstantial claim - Findingown data
The optimal configuration of DRG-LLaMA achieved a top-5 prediction accuracy of 86.5%.
“The optimal configuration, utilizing a 13B LLaMA model and a maximum input token size of 1024, achieved a top-1 prediction accuracy of 54.6%, a top-5 prediction accuracy of 86.5%, and a macro-F1 score of 0.361.”
Quote in the paperSubstantial claim - Findingown data
DRG-LLaMA-7B outperformed ClinicalBERT and CAML across all evaluation metrics in DRG prediction.
“This model surpassed the performance of prior leading models in DRG prediction”
- Also stated in: § DRG prediction as a single-label classification task · plain
“DRG-LLaMA consistently outperformed ClinicalBERT and CAML across all evaluation metrics”
Quote in the paperSubstantial claim - Also stated in: § DRG prediction as a single-label classification task · plain
- Findingown data
DRG-LLaMA's performance declined in less frequent DRGs.
“As expected, DRG-LLaMA ’s performance declined in less frequent DRGs”
Quote in the paperNot a substantial claim - Findingown data
In this study, the performance of both baseline models and DRG-LLaMA surpassed the outcomes reported in prior research.
“In our study, the performance of both the baseline models and DRG-LLaMA surpassed the outcomes reported in prior research13.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
- Findingreasoning
Implementing the rules culminated in a final DRG prediction accuracy close to single-label prediction (51.5% vs. 52.0%).
“Implementing such rules has culminated in a final DRG prediction accuracy close to single-label prediction (51.5% vs. 52.0%).”
Quote in the paperSubstantial claim - Findingown data
In experiments with a maximum input token limit up to 1024, the reported limitation of performance decline as input context expands was not encountered.
“In our constrained experiments conducted with a maximum input token limit up to 1024, we have yet to encounter this limitation.”
Quote in the paperNot a substantial claim - Interpretationown data
The LLM was able to achieve state-of-the-art performance via either classification setting.
“showing that the LLM was able to achieve state-of-the-art performance via either classification setting.”
Quote in the paperSubstantial claim - Interpretationown data
Surgical DRGs have distinct hospital courses that make them easier for the model to comprehend.
“which have distinct hospital courses that make them easier for the model to comprehend”
Quote in the paperNot a substantial claim - Interpretationown data
Predicting the principal diagnosis or procedure without considering CC/MCC is a significantly easier task on its own.
“This result suggests that predicting the principal diagnosis or procedure without considering CC/MCC is a significantly easier task on its own.”
Quote in the paperSubstantial claim - Interpretationown data
Other factors, such as the type of DRG, also affect prediction performance.
“However, other factors, such as the type of DRG, also affect prediction performance.”
Quote in the paperNot a substantial claim - Interpretationown data
Errors such as the unavailability of correct information required for DRG prediction underscore the limitations of solely relying on discharge summaries for DRG predictions.
“Furthermore, errors such as the unavailability of correct information required for DRG prediction underscore the limitations of solely relying on discharge summaries for DRG predictions.”
Quote in the paperSubstantial claim - Premisecitation
Centers for Medicare & Medicaid Service adjusts MS-DRG regulations annually, resulting in varying DRG assignments for identical conditions over time within the MIMIC-IV dataset.
“Centers for Medicare & Medicaid Service adjusts MS-DRG regulations annually, resulting in varying DRG assignments for identical conditions over time within the MIMIC-IV dataset28.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [28] Johnson, A. Question about DRG codes in MIMIC-IV. https://github.com/MIT-LCP/mimic-code/issues/1561 (2023).
Not checked in the free run.
- Interpretationreasoning
The enhanced performance of the models is plausibly predominantly linked to the strategic input data selection.
“Beyond the substantially larger training dataset employed in MIMIC-IV compared to MIMIC-III (236,192 vs. 17,815), it is plausible that this enhanced performance is predominantly linked to our strategic input data selection.”
Quote in the paperNot a substantial claim - Premisecitation
The MIMIC-IV dataset comprises 431,231 unique hospital admissions from 299,712 patients admitted to an ICU or the ED of the Beth Israel Deaconess Medical Center in Boston, Massachusetts.
“We conducted a study using the publicly available MIMIC-IV dataset, which comprises 431,231 unique hospital admissions from 299,712 patients admitted to an ICU or the ED of the Beth Israel Deaconess Medical Center in Boston, Massachusetts27.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [27] Johnson, A. E. et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci. Data 10, 1 (2023). source
Not checked in the free run.
- Interpretationreasoning
The unexpected CC/MCC results likely stem from the noisy nature of CC/MCC assignment.
“These unexpected results likely stem from the noisy nature of CC/MCC assignment.”
Quote in the paperSubstantial claim - Premisecitation
The study by Liu et al. included only clinical notes charted up to 48 hours post-admission or 48 hours after ICU admission.
“The study by Liu et al.13 included only clinical notes charted up to 48 hours post-admission or 48 hours after ICU admission.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
- Interpretationcitation
The selection of acute renal failure as the correct base DRG is guided by the DRG assignment rule, a factor beyond what is directly evident within the discharge summary.
“This selection is guided by the DRG assignment rule8, a factor extending beyond the scope of what is directly evident within the discharge summary.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [8] CMS. ICD-10-CM/PCS MS-DRG v34. 0 Definitions Manual. https://www.cms.gov/icd10m/version34-fullcode-cms/fullcode_cms/P0001.html (2016).
Not checked in the free run.
- May combine two claims. 22
- The quote only partly states this claim. 5
- Classified as probably not a substantial claim. 27
May combine two claims.
Findingown dataClinicalBERT achieved higher AUC and top-1 prediction accuracy but lower macro-averaged F1 score than CAML.
“When compared to CAML, ClinicalBERT achieved higher AUC and top-1 prediction accuracy but lower macro-averaged F1 score.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataDRG-LLaMA-7B achieved a macro-averaged F1 score of 0.327 on DRG prediction with a maximum input token size of 512.
“our DRG-LLaMA -7B model exhibited a noteworthy macro-averaged F1 score of 0.327”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe distribution of cases per DRG is imbalanced, with a median number of 124.5 in the training set.
“The distribution of cases per DRG is imbalanced, with a median number of 124.5 in the training set”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataDRG-LLaMA-7B achieved a macro-averaged AUC of 0.986 on DRG prediction with a maximum input token size of 512.
“a macro-averaged Area Under the Curve (AUC) of 0.986”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn manual error analysis of 10 samples, errors were categorized as erroneous CC/MCC (1/10), correct information needed for DRG prediction unavailable (1/10), difficulty in selecting correct base DRG (3/10), inadequate clinical concept extraction (4/10), and an isolated case of a plausible incorrect DRG label (1/10).
“Broadly, the identified errors were categorized as follows: erroneous CC/MCC (1/10), correct information needed for DRG prediction unavailable (1/10), difficulty in selecting correct base DRG (3/10), inadequate clinical concept extraction (4/10) and an isolated case of a plausible incorrect DRG label (1/10).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataThe training and testing set contains 738 and 723 unique DRG labels, respectively.
“The training and testing set contains 738 and 723 unique DRG labels, respectively.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataThe two-label approach's accuracy of 51.5% was comparable with the single-label approach's accuracy of 52.0% using the same base model.
“Notably, this performance was comparable with the accuracy attained in the single-label approach of 52.0% using the same base model”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataDRG-LLaMA-7B achieved a top-1 prediction accuracy of 52.0% on DRG prediction with a maximum input token size of 512.
“a top-1 prediction accuracy of 52.0%”
- Also stated in: § DRG prediction as a single-label classification task · plain
“The accuracy of top-1 and top-5 predictions achieved by our fine-tuned DRG-LLaMA -7B model was 52.0% and 84.8%, respectively.”
Quote in the paperSubstantial claim- Also stated in: § DRG prediction as a single-label classification task · plain
May combine two claims.
FindingreasoningThe top-1 accuracy for CC/MCC is 67.5%, similar to 67.8% for base DRG, despite the considerably smaller label count (5 labels in CC/MCC vs. 340 labels in base DRG).
“Surprisingly, the top-1 accuracy for CC/MCC stands at 67.5%, similar to 67.8% of the base DRG despite the considerably smaller label count (5 labels in CC/MCC vs. 340 labels in base DRG).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataMS-DRG version 34.0 comprises a total of 757 DRG codes, with 738 being represented in our dataset.
“This version comprises a total of 757 DRG codes, with 738 being represented in our dataset.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingreasoningIn Case 2 in Table 4, despite the discharge summary providing a more comprehensive discussion on gastrointestinal bleeding compared to acute renal failure, acute renal failure was deemed the correct base DRG.
“For instance, in Case 2 in Table 4, despite the discharge summary providing a more comprehensive discussion on gastrointestinal bleeding compared to acute renal failure, the latter was deemed the correct base DRG.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenDRG-LLaMA performance improved with the utilization of larger models and longer input context length.
“We also observed that the performance of DRG-LLaMA enhanced with the utilization of larger models and longer input context length.”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
FindingreasoningRules formulated in the DRG dissection phase and the inference phase cater to various split scenarios and improve accuracy.
“These rules cater to various split scenarios, thus improving accuracy.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone given90% of the data was allocated as training set while the rest 10% as testing set, and this partitioning was stratified on DRGs.
“90% of the data was allocated as training set while the rest 10% as testing set, and this partitioning was stratified on DRGs.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrevious studies have showcased advancements in DRGs classification accuracy through various machine-learning algorithms and deep neural networks.
“Previous studies have showcased advancements in DRGs classification accuracy through various machine-learning algorithms10 and deep neural networks11.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [10] Gartner, D., Kolisch, R., Neill, D. B. & Padman, R. Machine learning approaches for early DRG classification and resource allocation. INFORMS J. Comput. 27, 718–734 (2015). source
Not checked in the free run.
- cites [11] Islam, M.M., Li, G.H., Poly, T.N., Li, Y.C. Deepdrg: Performance of artificial intelligence model for real-time prediction of diagnosis-related groups. in Healthcare;9:1632MDPI (2021). source
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationEach distinct DRG code is delineated by a particular set of patient attributes, including principal diagnosis, specific secondary diagnoses, procedures, sex and discharge status.
“Each distinct DRG code is delineated by a particular set of patient attributes, including principal diagnosis, specific secondary diagnoses, procedures, sex and discharge status8.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [8] CMS. ICD-10-CM/PCS MS-DRG v34. 0 Definitions Manual. https://www.cms.gov/icd10m/version34-fullcode-cms/fullcode_cms/P0001.html (2016).
Not checked in the free run.
May combine two claims.
InterpretationcitationHigh AUC scores were obtained for all models due to the many infrequent DRG classes, resulting in high true negative predictions for all negative class predictions.
“High AUC scores were obtained for all models due to the many infrequent DRG classes, resulting in high true negative predictions for all negative class predictions13.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA deep learning-based NLP model leveraging CAML has been applied to predict DRGs based on clinical notes and yielded promising outcomes.
“More recently, a deep learning-based NLP model leveraging adjusted Convolutional Attention for Multi-Label Classification (CAML) has been applied to predict DRGs based on clinical notes and yielded promising outcomes12,13.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [12] Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., Eisenstein, J. Explainable prediction of medical codes from clinical text. arXiv preprint arXiv:1802.05695 (2018). source
Not checked in the free run.
- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown dataCertain errors, like inadequate clinical concept extraction, indicate the model's weaknesses.
“Certain errors, like inadequate clinical concept extraction, indicate the model’s weaknesses.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningOther errors, such as the difficulty in selecting the base DRG, likely stem from the intricacies of the DRG assignment rules.
“Other errors, such as the difficulty in selecting the base DRG, likely stem from the intricacies of the DRG assignment rules.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationDRGs involve multi-class classification, where one DRG code is assigned to each visit, in contrast to the multi-label classification of ICDs, where multiple codes may apply to a single visit.
“DRGs involve multi-class classification, where one DRG code is assigned to each visit, in contrast to the multi-label classification of ICDs, where multiple codes may apply to a single visit9.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9] Kaur, R., Ginige, J.A., Obst, O. AI-based ICD coding and classification approaches using discharge summaries: A systematic literature review. Expert Syst. Appl. 118997 (2022). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThere is no significant difference in the average word counts in the training vs. testing set (398 vs. 399; p = 0.51 from two-sided t-test).
“There is no significant difference in the average word counts in the training vs. testing set (398 vs. 399; p = 0.51 from two-sided t-test).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSince its inception by Medicare in 1983, DRG has served as the foundation for the inpatient prospective payment system within the United States.
“Since its inception by Medicare in 1983, DRG has served as the foundation for the inpatient prospective payment system within the United States7.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [7] Quinn, K. After the revolution: DRGs at age 30. Ann. Internal Med. 160, 426–429 (2014). source
Not checked in the free run.
May combine two claims.
PremisecitationIn the 34.0 version of the MS-DRG system, there are 154 three-way split DRGs, 44 two-way split DRGs with MCC/CC and no CC, 65 two-way split DRGs with MCC and CC/no CC, and 77 base DRGs with no splits.
“In the 34.0 version of the MS-DRG system, there are 154 three-way split DRGs, 44 two-way split DRGs with MCC/CC and no CC, 65 two-way split DRGs with MCC and CC/no CC, and 77 base DRGs with no splits (examples in Supplementary Note 1)8.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [8] CMS. ICD-10-CM/PCS MS-DRG v34. 0 Definitions Manual. https://www.cms.gov/icd10m/version34-fullcode-cms/fullcode_cms/P0001.html (2016).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe study was restricted by computational resource limitations, so experiments with the LLaMA model were only up to a parameter size of 13 billion.
“We were also restricted by computational resource limitations, so we could only experiment with the LLaMA model up to a parameter size of 13 billion.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSeveral groups have directed efforts toward fine-tuning LLaMA in the medical context, including ChatDoctor, HuaTuo, and PMC-LLaMA.
“Within the medical context, several groups have directed their efforts toward fine-tuning LLaMA. Notable examples among these are ChatDoctor (trained on authentic patient-physician dialogues), HuaTuo (fine-tuned with a Chinese medical knowledge graph), and PMC-LLaMA (fine-tuned on biomedical academic papers)22–24.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [22] Wang, H. et al. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975 (2023).
Not checked in the free run.
- cites [23] Yunxiang, L., Zihan, L., Kai, Z., Ruilong, D., You, Z. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070 (2023).
Not checked in the free run.
- cites [24] Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCAML model, exclusively utilizing clinical notes, surpassed the performance of a Long Short-Term Memory (LSTM) model using structured clinical variables.
“CAML model, exclusively utilizing clinical notes, surpassed the performance of a Long Short-Term Memory (LSTM) model using structured clinical variables13.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThese LLaMA-based models focused on medical question answering, yielding encouraging outcomes.
“These LLaMA-based models focused on medical question answering, yielding encouraging outcomes.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [22] Wang, H. et al. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975 (2023).
Not checked in the free run.
- cites [23] Yunxiang, L., Zihan, L., Kai, Z., Ruilong, D., You, Z. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070 (2023).
Not checked in the free run.
- cites [24] Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 (2023).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningAn extensive hyperparameter search was not performed.
“Unfortunately, we couldn’t perform an extensive hyperparameter search.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningAssigning a label of 'not applicable' when a DRG code does not explicitly describe CC/MCC status might bring noisy signals for models to learn, but was found better than assigning to an alternative label such as 'without CC/MCC'.
“We realize such classification might bring in noisy signals for models to learn (as a patient with “transient ischemia” can indeed has CC/MCC), but found it better than assigning to an alternative label such as “without CC/MCC” which would be more erroneous.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationClinicalBERT was built upon BioBERT, a domain-specific BERT model pre-trained on PubMed abstracts and full-text articles from PubMed Central.
“ClinicalBERT was built upon BioBERT, a domain-specific BERT model pre-trained on PubMed abstracts and full-text articles from PubMed Central16.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [16] Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationWhen compared with ClinicalBERT, CAML provided improved F1 scores but lower AUC.
“When compared with ClinicalBERT, CAML provided improved F1 scores but lower AUC13,15.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
- cites [15] Alsentzer, E. et al. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323 (2019). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCAML model exhibited superior ability to predict DRGs.
“More recently, CAML model exhibited superior ability to predict DRGs13.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenThe MIMIC-IV training dataset is substantially larger than the MIMIC-III dataset (236,192 vs. 17,815).
“Beyond the substantially larger training dataset employed in MIMIC-IV compared to MIMIC-III (236,192 vs. 17,815)”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningA hospital stay with the DRG code 'pulmonary edema and respiratory failure' may truly contain MCC, but the MCC would not be labeled as positive in the training set.
“Therefore, a hospital stay with this DRG code may truly contain MCC, but the MCC would not be labeled as positive in the training set.”
Quote in the paperSubstantial claim- Rests on an assumption: The DRG code 'pulmonary edema and respiratory failure' does not have a CC/MCC split.
May combine two claims.
PremisecitationWithout domain-specific fine-tuning or specialized prompt crafting, GPT-4 exceeded the passing score on USMLE by over 20 points and set a new state-of-the-art.
“Without domain-specific fine-tuning or specialized prompt crafting, GPT-4 exceeded the passing score on USMLE by over 20 points and set a new state-of-the-art3.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [3] Nori, H., King, N., McKinney, S.M., Carignan, D., Horvitz, E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationInstruction-following models fine-tuned from LLaMA such as Alpaca and Vicuna exhibit performance on par with GPT-3.5.
“Instruction-following models fine-tuned from LLaMA such as Alpaca20 and Vicuna21, exhibit performance on par with GPT-3.5.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [20] Taori, R. et al. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca (2023).
Not checked in the free run.
- cites [21] Chiang, W.L. et al. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna (2023).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationClinicalBERT performed further pre-training of BioBERT using 2 million clinical notes from MIMIC-III.
“ClinicalBERT performed further pre-training of BioBERT using 2 million clinical notes from MIMIC-III34.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [34] Johnson, A. E. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 1–9 (2016). source
Not checked in the free run.
May combine two claims.
Study design and conductown dataThe study was limited by the constraints of the MIMIC-IV dataset and could only use discharge summaries as input data, which are only available after the patient is discharged from the hospital.
“We were limited by the constraints of the MIMIC-IV dataset and could only use discharge summaries as input data, which are only available after the patient is discharged from the hospital.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenAn internal medicine physician manually reviewed all DRG codes from step 2 and assigned them to the most appropriate MS-DRG v.34.0 codes; a domain expert specializing in inpatient Clinical Documentation Integrity independently verified the accuracy of the code assignments.
“An internal medicine physician manually reviewed all DRG codes from step 2, and assigned these codes to the most appropriate MS-DRG v.34.0 codes if applicable. Subsequently, a domain expert specializing in inpatient Clinical Documentation Integrity (CDI) assessed the conversion table and independently verified the accuracy of the code assignments.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrevious studies have underscored the effectiveness of employing diverse machine learning algorithms and deep neural networks for DRG prediction within healthcare systems outside the United States.
“Previous studies have underscored the effectiveness of employing diverse machine learning algorithms and deep neural networks for DRG prediction within healthcare systems outside the United States10,11.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [10] Gartner, D., Kolisch, R., Neill, D. B. & Padman, R. Machine learning approaches for early DRG classification and resource allocation. INFORMS J. Comput. 27, 718–734 (2015). source
Not checked in the free run.
- cites [11] Islam, M.M., Li, G.H., Poly, T.N., Li, Y.C. Deepdrg: Performance of artificial intelligence model for real-time prediction of diagnosis-related groups. in Healthcare;9:1632MDPI (2021). source
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThe results highlight the potential of adapting LLMs for medical purposes, particularly in predicting DRGs.
“The results presented in this study highlight the potential of adapting LLMs for medical purposes, particularly in predicting DRGs.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationBERT-based models have a maximum input length of 512 tokens, whereas CAML can handle longer context.
“While BERT-based models have a maximum input length of 512 tokens, CAML has the flexibility to handle longer context13,19.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [13] Liu, J., Capurro, D., Nguyen, A. & Verspoor, K. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ Digi.l Med. 4, 103 (2021). source
Not checked in the free run.
- cites [19] Devlin, J., Chang, M.W., Lee, K., Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThese studies focused on using structured data as input variables instead of clinical text.
“These studies focused on using structured data as input variables instead of clinical text.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [10] Gartner, D., Kolisch, R., Neill, D. B. & Padman, R. Machine learning approaches for early DRG classification and resource allocation. INFORMS J. Comput. 27, 718–734 (2015). source
Not checked in the free run.
- cites [11] Islam, M.M., Li, G.H., Poly, T.N., Li, Y.C. Deepdrg: Performance of artificial intelligence model for real-time prediction of diagnosis-related groups. in Healthcare;9:1632MDPI (2021). source
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationLanguage models based on the transformer architecture, either pretrained or fine-tuned using biomedical corpora, have demonstrated efficacy across a spectrum of NLP benchmarks within the biomedical realm.
“Language models based on the transformer architecture, either pretrained or fine-tuned using biomedical corpora, have demonstrated efficacy across a spectrum of NLP benchmarks within the biomedical realm16–18.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [16] Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020). source
Not checked in the free run.
- cites [17] Huang, K., Altosaar, J., Ranganath, R. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342 (2019).
Not checked in the free run.
- cites [18] Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transact. Comput. for Healthcare (HEALTH) 3, 1–23 (2021).
Not checked in the free run.
The paper relies on this without support.
The DRG assignment process is inefficient.
“but its assignment process is inefficient”
The paper relies on this without support.
Traditionally, the assignment of DRGs constitutes a labor-intensive manual endeavor undertaken by coding specialists, typically subsequent to a patient’s discharge.
“Traditionally, the assignment of DRGs constitutes a labor-intensive manual endeavor undertaken by coding specialists, typically subsequent to a patient’s discharge.”
The paper relies on this without support.
A pressing interest exists in the accurate early prediction of DRGs during a patient’s hospitalization.
“a pressing interest exists in the accurate early prediction of DRGs during a patient’s hospitalization”
The paper relies on this without support.
Accurate early DRG prediction is vital for efficacious resource planning and allocation.
“This prediction is vital for efficacious resource planning and allocation.”
The paper relies on this without support.
In MIMIC-III, a large portion of records during the 48-hour window comprises nursing and radiology notes, potentially lacking pivotal admission History of Present Illness (HPI) notes.
“In the MIMIC-III database, a large portion of records during this time window comprises nursing and radiology notes, potentially lacking the pivotal admission History of Present Illness (HPI) notes.”
The paper relies on this without support.
The DRG code 'pulmonary edema and respiratory failure' does not have a CC/MCC split.
“For instance, the DRG code “pulmonary edema and respiratory failure” does not have a CC/MCC split.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can serve as classifiers in clinical prediction tasks and have achieved outstanding performance.
“When provided with clinical text inputs, LLMs can also serve as classifiers in clinical prediction tasks and have achieved outstanding performance31–33.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Longer input context does not improve every metric
MinorFor the 7B model, increasing the maximum input length from 512 to 1024 tokens leaves both macro-AUC (0.986) and micro-AUC (0.994) unchanged, contrary to the claim of improvement in all evaluation metrics.
“We investigated DRG-LLaMA ’s performance across varying model sizes and input context lengths (Table 2), observing a consistent improvement in all evaluation metrics with larger models and longer input contexts, measured in maximum token numbers.”
“| 7B | 1024 | 0.346 (0.004) | 0.539 (0.003) | 0.861 (0.002) | 0.923 (0.001) | 0.986 (0.001) | 0.994 (0.000) | | | 512 | 0.327 (0.004) | 0.520 (0.003) | 0.848 (0.002) | 0.912 (0.002) | 0.986 (0.001) | 0.994 (0.000) |”
Cites
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3× · see citing works
“Beyond the substantially larger training dataset employed in MIMIC-IV compared to MIMIC-III (236,192 vs. 17,815), it is plausible that this enhanced performance is predominantly linked to our strategic input data selection.”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2× · see citing works
“…pretrained or fine-tuned using biomedical corpora, have demonstrated efficacy across a spectrum of NLP benchmarks within the biomedical realm (Lee et al. 2020; Huang, Altosaar, and Ranganath 2020; Gu Toward deploying a local LLM, we used LLaMA, a robust and openly accessible foundational LLM…”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2× · see citing works
“2020) and InstructGPT (Ouyang et al. 2022), has brought about a transformative shift in the landscape of Natural Language Processing (NLP).”
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401 · cited 1× · see citing works
“We conducted a study using the publicly available MIMIC-IV dataset, which comprises 431,231 unique hospital admissions from 299,712 patients admitted to an ICU or the ED of the Beth Israel Deaconess Medical Center in Boston, Massachusetts (Johnson et al. 2023).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 2× · see citing works
“Large language model context: Language models based on the transformer architecture, either pretrained or fine-tuned using biomedical corpora, have demonstrated efficacy across a spectrum of NLP benchmarks within the biomedical realm (Lee et al. 2020; Huang, Altosaar, and Ranganath 2020; Gu Toward deploying a local LL…”
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 2× · see citing works
“These LLaMA-based models focused on medical question answering, yielding encouraging outcomes.”
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 1× · see citing works
“Interestingly, a recent study revealed that the optimal performance of LLMs is attained when pertinent information is positioned at either the beginning or the end of the input context, with a decline as the input context expands (Liu et al. 2023).”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · cited 2× · see citing works
“These LLaMA-based models focused on medical question answering, yielding encouraging outcomes.”
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S. B., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 616 · see citing works
Explainable Prediction of Medical Codes from Clinical Text
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1100 · Cited by 597 · cited 3× · see citing works
“As baseline models for benchmarking, We selected CAML (Mullenbach et al. 2018; Liu et al. 2021) and ClinicalBERT (Alsentzer et al. 2019).”
Self-citationLi, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305 · see citing works
DocBERT: BERT for Document ClassificationPreprint
Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08398 · Cited by 286 · see citing works
AI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · cited 1× · see citing works
“Nonetheless, the medical domain introduces elevated concerns regarding safety and privacy, necessitating detailed analysis regarding the performance and limitations of LLMs to address the inherent risks such as hallucination, bias, and reasoning deficiencies (Au Yeung et al. 2023).”
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115 · see citing works
Self-citationMachine Learning Approaches for Early DRG Classification and Resource Allocation
Gartner, D., Kolisch, R., Neill, D. B., & Padman, R. (2015) · INFORMS Journal on Computing · doi:10.1287/ijoc.2015.0655 · Cited by 58 · cited 4× · see citing works
“Previous studies have underscored the effectiveness of employing diverse machine learning algorithms and deep neural networks for DRG prediction within healthcare systems outside the United States (Gartner et al. 2015; Islam et al. 2021).”
InfluentialLiu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00474-9 · Cited by 50 · cited 12× · see citing works
“As baseline models for benchmarking, We selected CAML (Mullenbach et al. 2018; Liu et al. 2021) and ClinicalBERT (Alsentzer et al. 2019).”
InfluentialKaur, R., Ginige, J. A., & Obst, O. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118997 · Cited by 48 · cited 1× · see citing works
“This distinction stems from differences in the nature of the task: DRGs involve multi-class classification, where one DRG code is assigned to each visit, in contrast to the multi-label classification of ICDs, where multiple codes may apply to a single visit (Kaur, Ginige, and Obst 2022).”
PMC-LLaMA: Towards Building Open-source Language Models for MedicinePreprint
Wu, C., Lin, W., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14454 · Cited by 48 · see citing works
After the Revolution: DRGs at Age 30
Quinn, K. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2115 · Cited by 47 · cited 3× · see citing works
“Previous studies have underscored the effectiveness of employing diverse machine learning algorithms and deep neural networks for DRG prediction within healthcare systems outside the United States (Gartner et al. 2015; Islam et al. 2021).”
Islam, M. M., Li, G.-H., Poly, T. N., & Li, Y.-C. (. (2021) · Healthcare · doi:10.3390/healthcare9121632 · Cited by 19 · cited 4× · see citing works
“Previous studies have underscored the effectiveness of employing diverse machine learning algorithms and deep neural networks for DRG prediction within healthcare systems outside the United States (Gartner et al. 2015; Islam et al. 2021).”
InfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- Brown, T. et al. Language models are few-shot learners. Adv. Neural Inform. Process. Syst. 33, 1877–1901 (2020).
- Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
Cited by
Sorted by citations
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662 · see citing works
Privacy-preserving large language models for structured medical information retrieval
Wiest, I. C., Ferber, D., Zhu, J., van Treeck, M., Meyer, S. K., Juglan, R., Carrero, Z. I., Paech, D., Kleesiek, J., Ebert, M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01233-2 · Cited by 170 · see citing works
“Llamahasbeen previouslysuccessfulintaskslikeDRGpredictionandtestedforICDcode extractionfromclinicalnotes 33,34 .”
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94 · see citing works
Zero shot health trajectory prediction using transformer
Renc, P., Jia, Y., Samir, A. E., Wąs, J., Li, Q., Bates, D. W., & Sitek, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01235-0 · Cited by 67 · see citing works
“%) in 28,932 hospitalizations using our methodology, a signi fi cant improvement over the 52% reported by Wang et al. 13 , who explored DRG estimationusing LLMsfrom discharge notes.”
Ong, J. C. L., Jin, L., Elangovan, K., Lim, G. Y. S., Lim, D. Y. Z., Sng, G. G. R., Ke, Y. H., Tung, J. Y. M., Zhong, R. J., Koh, C. M. Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102323 · Cited by 58 · see citing works
Toward Clinical Generative AI: Conceptual Framework
Bragazzi, N. L., & Garbarino, S. (2024) · JMIR AI · doi:10.2196/55957 · Cited by 64 · see citing works
“Clinically oriented LLMs [19-25] such as ClinicalBERT, BlueBERT, CAML, DRG-LLaMA, GatorTronGPT, or PaLM have shown impressive capabilities, yet their application in clinical settings faces stringent requirements.”
BackgroundBuess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 71 · see citing works
“…[36] NER BioBERT [26] NER, QA BioGPT [37] Classification, QA BioMistral [27] QA ChatDoctor [4] Dialogue ChestXRayBERT [3] Summarization DRG-LLaMA [38] Classification GatorTron [39] QA HuatuoGPT [30] Dialogue HuatuoGPT-o1 [30] Dialogue Johnson et al. [40] Deidentification Kresevic et al. [41]…”
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738 · see citing works
Wang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025) · Journal of Medical Internet Research · doi:10.2196/64486 · Cited by 51 · see citing works
“At present, language models based on transformer architecture, whether pretrained or fine-tuned using biomedical corpora, have been proven effective in a series of natural language processing benchmarks in the biomedical field [183].”
Fan, D., Liu, M., Shao, Y., Yang, L., Liu, Y., Zhang, Y., Ren, Y., & Wang, Z. (2025) · Engineering · doi:10.1016/j.eng.2025.12.019 · Cited by 42 · see citing works
Shashikumar, S. P., Mohammadi, S., Krishnamoorthy, R., Patel, A., Wardi, G., Ahn, J. C., Singh, K., Aronoff-Spencer, E. S., & Nemati, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01689-w · Cited by 41 · see citing works
“Initial investigations have demonstrated successful applications of LLMs in identifying social determinants of health 30 , assigning diagnosis-related groups 31 , answering medical questions 32 , drafting clinical notes 33 , and drafting clinician responses to patient inquiries 34,35 .”
A dataset and benchmark for hospital course summarization with adapted large language models
Aali, A., Van Veen, D., Arefeen, Y. I., Hom, J., Bluethgen, C., Reis, E. P., Gatidis, S., Clifford, N., Daws, J., Tehrani, A. S., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae312 · Cited by 32 · see citing works
“Other works [40, 41] further explore domain adaptation of LLMs in the clinical domain through fine-tuning for improved performance in medical tasks.”
Pashangpour, S., & Nejat, G. (2024) · Robotics · doi:10.3390/robotics13080112 · Cited by 37 · see citing works
“Table 1 presents an overview of five specialized LLMs designed for healthcare: (1) BiomedBERT [41], (2) MED-PaLM 2 [42], (3) DRG-LLaMA [43], (4) GPT2-BioPt [44], and (5) Clinical-T5 [45].”
MethodLarge Language Models for Health Care Text Classification: Systematic Review
Sakai, H., & Lam, S. S. (2025) · JMIR AI · doi:10.2196/79202 · Cited by 43 · see citing works
“Wang et al [43] developed DRG (Diagnosis-Related Group)-LLaMA, a fine-tuned version of the LLaMA LM, to predict diagnosis-related groups from clinical discharge summaries in the MIMIC-IV dataset.”
Du, X.-S., Zhou, Z., Wang, Y.-F., Chuang, Y.-W., Li, Y.-M., Yang, R., Zhang, W., Wang, X., Chen, X.-Y., Guan, H., et al. (2024) · medRxiv · doi:10.1101/2024.08.11.24311828 · Cited by 36 · see citing works
Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R., Hong, P., Bates, D. W., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106091 · Cited by 26 · see citing works
Liu, C., & Chou, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105891 · Cited by 23 · see citing works
Wang, Y., Huang, Y., Nimma, I. R., et al. (2024) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.16561 · Cited by 20 · see citing works
“19,20 This promising area merits further investigation.”
Graph retrieval augmented large language models for facial phenotype associated rare genetic disease
Song, J., Xu, Z., He, M., Feng, J., & Shen, B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01955-x · Cited by 19 · see citing works
“…due to their impressive comprehension, inference, and generative capabilities, and have begun to show great potential in the fi eld of biomedical, particularly in the areas of medical diagnostics 10 , mental health 11 , biological materials 12 , hospital management 13 , and medical education 14 .”
A perspective for adapting generalist AI to specialized medical AI applications and their challenges
Wang, Z., Wang, H., Danek, B. P., Li, Y., Mack, C. D., Arbuckle, L., Biswal, D., Poon, H., Wang, Y., Rajpurkar, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01789-7 · Cited by 25 · see citing works
“Additionally, we discuss some potential improvements of those use cases.”
Self-citationLarge Language Model Symptom Identification From Clinical Text: Multicenter Study
McMurry, A., Phelan, D., Dixon, B. E., Geva, A., Gottlieb, D., Jones, J. R., Terry, M., Taylor, D. E., Callaway, H., Manoharan, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72984 · Cited by 20 · see citing works
“The context [18,21,28] surrounding infectious respiratory diseases includes symptoms pertaining to acute infections, noninfectious conditions, treatment side effects [6], indications for treatment, or patient instructions (eg, “Use albuterol inhaler as needed for difficulty breathing”).”
Zheng, Y., Yan, Y., Chen, S., et al. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1501408 · Cited by 15 · see citing works
Shi, B., Chen, L., Pang, S., Wang, Y., Wang, S., Li, F., Zhao, W., Guo, P., Zhang, L., Fan, C., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/67253 · Cited by 15 · see citing works
“It proposed the DRG-Llama-7B model, which achieved an AUROC of 0.986 [24].”
InfluentialMedAdapter: Efficient Test-Time Adaptation of Large Language Models Towards Medical Reasoning
Shi, W., Xu, R., Zhuang, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1244 · Cited by 13 · see citing works
Huang, Y., Ao, T., Peng, Z. M., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35199 · Cited by 11 · see citing works
Extracting Triplets from Domain-specific Texts using Fine-Tuned LLaMA Models: A Comparative Study
Veena, G., Rajendran, A., & Gupta, D. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.630 · Cited by 11 · see citing works
From Single-Cancer to Pan-Cancer Prognosis
Zhang, B., Li, S., Jian, J., Ren, X., Zhao, Z., Guo, L., Su, F., Meng, Z., & Zhao, Z. (2025) · American Journal Of Pathology · doi:10.1016/j.ajpath.2025.06.006 · Cited by 11 · see citing works
Li, Y., Zhan, Y., Liang, M., Zhang, Y., & Liang, J. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113999 · Cited by 10 · see citing works
Huang, Y., Ao, T., Peng, Z. M., et al. (2025) · Scientific Reports · doi:10.1038/s41598-024-84385-1 · Cited by 9 · see citing works
The Development Landscape of Large Language Models for Biomedical Applications
Cao, Z., Keloth, V. K., Xie, Q., Qian, L., Liu, Y., Wang, Y., Shi, R., Zhou, W., Yang, G. F., Zhang, J., et al. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102224-074736 · Cited by 14 · see citing works
Ojemann, W., Xie, K., Liu, K., Chang, E., Roth, D., Litt, B., & Ellis, C. A. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00198-5 · Cited by 9 · see citing works
“Wang et al. compared a fine-tuned ClinicalBERT to fine-tuned Llama models and found the latter to be better at classifying diagnosis-related groups [28].”
In-Storage Acceleration of Retrieval Augmented Generation as a Service
Mahapatra, R., Santhanam, H., Priebe, C., Xu, H., & Esmaeilzadeh, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731032 · Cited by 10 · see citing works
Villegas-Ch, W. E., Govea, J., & Gutierrez, R. (2024) · IEEE Access · doi:10.1109/access.2024.3518952 · Cited by 10 · see citing works
Liu, J., Nguyen, A. N., Daniel Capurro, N., & Verspoor, K. M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3574254 · Cited by 9 · see citing works
“Consequently, it has become increasingly common to apply Natural Language Processing (NLP) methods to train prediction models based on clinical text [14], [15], [16], leading to various applications in risk assessment [17], [18], early diagnosis [19], [20], and resource prediction [21], [22].”
Chen, Y.-H., Ruan, S.-J., & Chen, P. (2025) · Journal of Medical Internet Research · doi:10.2196/75052 · Cited by 11 · see citing works
“The model achieved an area under the curve of 0.986 and outperformed existing methods like ClinicalBERT, showcasing its utility in handling complex health care–related classification tasks [19].”
InfluentialSulaiman, M. H., Muda, N., & Razak, F. A. (2025) · JMIR Formative Research · doi:10.2196/69075 · Cited by 7 · see citing works
“For example, recent studies were conducted to see the impact of LLM in drug discovery [26], extraction of medical notes [27], prediction of diagnosis-related groups [28], and diagnostics [29].”
Ting, D. S. W., Ong, J. C. L., Jin, L., Elangovan, K., Lim, G. Y. S., Lim, D. Y. Z., Sng, G. G. R., Ke, Y., Tung, J. Y. M., Zhong, R. J., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4023142/v1 · Cited by 7 · see citing works
Exploring large language models for summarizing and interpreting an online brain tumor support forum
Muasher-Kerwin, C., Hughes, M. C., Foster, M. L., Azher, I. A., & Alhoori, H. (2025) · Digital Health · doi:10.1177/20552076251337345 · Cited by 7 · see citing works
“13 LLMs have the potential to offer signi fi cant bene fi ts in environments such as online community forums, where patients and caregivers share experiences, seek support, and discuss treatment-related challenges.”
Li, R., Mao, S., Zhu, C., Yang, Y., Tan, C., Li, L., Mu, X., Liu, H., & Yang, Y. (2025) · Journal of Medical Internet Research · doi:10.2196/72638 · Cited by 6 · see citing works
“More recently, the advent of large language models (LLMs), including ChatGPT (OpenAI), BARD (Google), and LLaMA (Meta AI), has provided researchers with novel tools for analyzing complex medical data [7-10], particularly clinical text.”
Gao, H., Wang, K., Yuan, Y., Wang, Y., Liu, Q., Wang, Y., Sun, J. Q., Wang, W., Wang, H., Zhou, S., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-07649-4 · Cited by 8 · see citing works
Self-citationLiu, X., Yeung, W. L. K., Chen, Z., Hao, S., Yang, Z., Sun, X., Liu, C., Mao, Z., Yan, M., Yan, W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01875-w · Cited by 6 · see citing works
“Clinical language models (LM), such as ClinicalBERT, PubMedBERT, and DRG-LLaMA, which were pretrained on biomedical and medical corpora, have shown increased pro fi ciency and ability when fi ne-tuned for tasks such as predicting in-hospital mortality and length of hospital stay 20,21 .”
Heart failure prognosis prediction: Let’s start with the MDL-HFP model
Ma, H., Li, D., Fu, J., Zhao, G., & Zhao, J. (2024) · Information Systems · doi:10.1016/j.is.2024.102408 · Cited by 6 · see citing works
Huang, Y., Ao, T., Peng, Z. M., et al. (2024) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2024.1458238 · Cited by 6 · see citing works
Large Language Model (LLM) Support for Preliminary Consultation in Healthcare
Fernando, S. P. A. A., & Wickramaarachchi, D. (2025) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc64760.2025.10963287 · Cited by 5 · see citing works
“In order to help with early detection and the diagnosis of rare diseases, LLMs trained on datasets like as MIMIC-III are excellent diagnosticians who find patterns in unstructured data [3] [13].”
Tao, Y., Shen, K., Chen, Y., et al. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1567838 · Cited by 5 · see citing works
“…presents a formidable challenge to national healthcare systems (4), prompting countries to seek methods for rationalizing healthcare costs (5), For instance, Germany’s German Diagnosis-Related Groups (G-DRG) System, the Accountable Care Organizations (ACOs) model in the United States,…”
Jiang, R., Chen, B., Dong, Z., Zeng, X., You, H., Li, Y., Deng, Y., Mu, G., Wang, J., Huang, L., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02569-7 · Cited by 6 · see citing works
“MLLMs are arti fi cial intelligence algorithms based on large language models that can integrate and correlate more modal data to achieve cross-modal tasks 32 .”
Huang, Y., Ao, T., Peng, Z. M., et al. (2024) · BMC Cardiovascular Disorders · doi:10.1186/s12872-024-04258-3 · Cited by 5 · see citing works
Papageorgiou, P. S., Christodoulou, R. C., Pitsillos, R., Petrou, V., Vamvouras, G., Kormentza, E. V., Papagelopoulos, P. J., & Georgiou, M. F. (2025) · Applied Sciences · doi:10.3390/app15169005 · Cited by 8 · see citing works
Tian, W., Zhang, S., Gao, Y., et al. (2024) · Technology and Health Care · doi:10.3233/thc-240284 · Cited by 4 · see citing works
Large Language Model Symptom Identification from Clinical Text: A Multi-Center StudyPreprint
McMurry, A., Phelan, D., Dixon, B. E., Geva, A., Gottlieb, D., Jones, J. R., Terry, M., Taylor, D. A., Callaway, H. G., Mahoharan, S., et al. (2024) · medRxiv · doi:10.1101/2024.12.16.24319044 · Cited by 4 · see citing works
Feng, Y. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03151-z · Cited by 3 · see citing works
“The authors reported that their DRG-LLaMA [29] achieved a macro-F1 score of 0.327 on all DRGs, surpassing previous leading models in DRG prediction, including ClinicalBERT.”
InfluentialGhosh, A., Jain, R., Jhangra, A., Saha, S., & Jatowt, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70045 · Cited by 5 · see citing works
Bala, I., Sharma, E., & Mitchell, L. (2025) · Artificial Intelligence Surgery · doi:10.20517/ais.2025.44 · Cited by 3 · see citing works
“We are a club none of us wanted to join”: exploring brain tumor online discussion forum content
Muasher-Kerwin, C., Hughes, M. C., Foster, M. L., & Econie, S. M. (2025) · Supportive Care in Cancer · doi:10.1007/s00520-025-09545-z · Cited by 3 · see citing works
Kang, B., Lee, S., Bae, H., & Kim, C.-E. (2024) · 동의생리병리학회지 · doi:10.15188/kjopp.2024.04.38.2.49 · Cited by 3 · see citing works
MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“Whenprovidedwithclinicaltextinputs,LLMscanalsoserve as classi fi ers in clinical prediction tasks and have achieved outstanding performance 31 – 33 .”
A tree-structured multiobjective optimization framework for constructing diagnosis-related groups
Cai, G., Zeng, Z., Wan, M., Liu, N., Wang, Y., & Niu, B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02038-7 · Cited by 2 · see citing works
“This method has been proven to be effective in incentivizing healthcare institutions to optimize resource allocation, reduce unnecessary lengths of stay and medical expenses, and improve the quality of medical services 5 .”
Wang, Y., Song, Y., Siu, R. K., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag008 · Cited by 2 · see citing works
Cheema, M. N., Nazir, A., Harmanci, A. O., Harmanci, A. O., Cheema, Y., Masood, S., & Khokhar, F. A. (2026) · The Visual Computer · doi:10.1007/s00371-026-04385-2 · Cited by 2 · see citing works
“Introducing Large Language Models (LLMs) into radiogenomics offers a new paradigm for advancing toward precision medicine [6].”
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z.-Z., Yang, Y.-F., Wang, Z., Xiong, G.-Z., Lai, P.-T., Zhu, Q.-Q., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 3 · see citing works
Self-citationElyas, O. A., Hashim, H. W. A., & Williams, J. R. (2025) · SPE Western Regional Meeting · doi:10.2118/224128-ms · Cited by 2 · see citing works
(2024) · Journal of System and Management Sciences · doi:10.33168/jsms.2024.1015 · Cited by 1 · see citing works
Li, X., Huang, Y., & Sun, L. (2025) · Respiratory Medicine · doi:10.1016/j.rmed.2025.108489 · Cited by 1 · see citing works
VIS-Llama: Automated Visualization of Food Cold Chain Data Leveraging the Large Language Model
Wang, L., Chen, Y., Lv, C., et al. (2025) · doi:10.1109/ijcnn64981.2025.11228089 · Cited by 1 · see citing works
Huang, Y., Ao, T., Hu, M., & Zhen, P. (2026) · Digital Health · doi:10.1177/20552076261421140 · Cited by 1 · see citing works
Dong, X., Huang, Y., Hu, M., et al. (2026) · Medicine · doi:10.1097/md.0000000000047988 · Cited by 1 · see citing works
“[11,12] Researchers Xiao Dong (Certification ID: 63109170) and Yingxiu Huang (Certification ID: 56513391) were granted access after completing the required training modules.”
Large Language Models for Clinical Narrative Processing: Methods, Applications, and Challenges
Livieratos, A., Lin, J., Chasani, P., Gaga, M., Fousekis, F. S., Gogos, C. A., Akinosoglou, K., Katsanos, K. H., & Gamalo, M. (2026) · Methods and Protocols · doi:10.3390/mps9030069 · Cited by 1 · see citing works
“[61,62,68–70 , 75 ].”
Lu, A.-T., Liou, C.-S., Lai, C.-H., Shain, B.-T., Li, M.-T., Sun, C.-Y., Kao, H.-Y., Dai, H.-J., & Tsai, M. (2024) · JMIR Human Factors · doi:10.2196/59961 · Cited by 1 · see citing works
“In this context, AI-assisted coding emerges as a powerful tool [24].”
Bhardwaj, S., Joshi, R. C., Tiwari, V., et al. (2025) · doi:10.1109/incacct65424.2025.11011436 · Cited by 1 · see citing works
“Models, e.g., LLAMA [11], RoBERTa [12] and BERT [13], show an excellent ability to learn context-based information from large scale data sets.”
Wu, M., Lin, H., Lyu, X., Zhang, C., & Yu, S. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385316
“The DRG-LLaMA [14] endows us with the awareness that with proper training and adjustments, LLM can effectively automate the classification of diagnosis-related groups in hospitals.”
MethodFeng, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-5750190/v1
Data Augmentation Techniques for Chinese Disease Name Normalization
Cui, W., Fu, X., Liu, S., Gu, M., Liu, X., Wu, J., & King, I. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821986
“…names play a pivotal role in modern intelligent healthcare systems as it is involved in diverse tasks such as intelligent consultation [1], auxiliary diagnosis [2], [3], automated International Classification of Diseases (ICD) coding [4]–[6], Diagnosis-Related Groups prediction [7], [8], etc.”
Albassam, D., Cross, A., & Zhai, C. (2025) · doi:10.2196/preprints.75979
Gan, L., Gao, X., Huang, Y., & Tan, J. (2025) · Journal of medicine and health science. · doi:10.62517/jmhs.202505107
Innovative artificial intelligence for practice management in medical healthcare
Żurek, D., Corizzo, R., & Kołtowski, Ł. (2025) · European Heart Journal · doi:10.1093/eurheartj/ehaf329
Autonomous LLMs for Healthcare Diagnoses
Ahmad, H. F., & Shafqat, S. (2025) · IEEE International Symposium on Autonomous Decentralized Systems (ISADS) · doi:10.1109/isads66912.2025.00015
Comparative models on low multiplier DRG classification for advanced lung cancer
Yu, M., & Zhang, Y. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1614938
“Their results indicated that DRG-LLaMA outperformed previously established models in DRG prediction accuracy (15).”
Wang, L., Tian, F., Li, F., Wu, M., Hou, L., Wang, J., Zhao, J., Feng, X., Li, C., Wang, X., et al. (2025) · EBioMedicine · doi:10.1016/j.ebiom.2025.106060 · Cited by 1 · see citing works
“31 LoRA decomposes the large weight matrices of transformer self-attention modules into two low-rank matrices (rank = 16, scaling factor = 8 in our experiments), introducing only a small number of train-able parameters while preserving the pretrained backbone.”
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Qu, J., Huang, Y., & Shen, L. (2025) · Medicine · doi:10.1097/md.0000000000046644
“[16,17] After successfully completing the online training and exam (Certificate ID: 56513391), Yingxiu Huang, one of the authors, was granted access to the database.”
Validation of 13,102 ICD-10-CM Codes Using a Large Language Model-Based SystemPreprint
Wang, Y., Song, Y., Siu, R. K., et al. (2025) · medRxiv · doi:10.64898/2025.12.30.25343244
Phadtare, A. S., Kulkarni, N. G., & Devare, M. H. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-025-00687-y · Cited by 1 · see citing works
“Wang et al. (2024) further optimised the complex big language model, DRG-LLaMA, for clinical notes by using low-rank adaptation optimization to a large dataset of discharge summaries.”
Huang, Y., Ao, T., Hu, M., et al. (2026) · Medicine · doi:10.1097/md.0000000000047798
Zhao, Y., Madill, E. S., Kitessa, A., Castle, B., Katre, A. R., & Chitnis, T. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00095
“More lightweight, open-source models like LLaMA [15] have since been adapted for the clinical domain, demonstrating strong zero-shot capabilities in prognosis prediction and guideline adherence extraction [16]–[19].”
Yang, J., Yang, S., Shao, Z., Chen, T., Shen, H., Zhou, P., Xia, B., Lei, X., Wang, L., Xue, D., et al. (2026) · Burns & Trauma · doi:10.1093/burnst/tkag026
“Diagnosis -related group (DRG)–large language model meta AI (LLaMA) successfully classifies billing groups with acceptable accurac y, streamlining reimbursement and resource allocation [ 129].”
Li, M., Wang, H.-B., Wang, H., & Xu, Z. (2026) · Medicine · doi:10.1097/md.0000000000048296 · Cited by 1 · see citing works
“[14,23,24] The study followed the Declaration of Helsinki.”
Self-citationMedGellan: LLM-Generated Medical Guidance to Support Physicians in Diagnosis
Banerjee, D., Sayin, B., Teso, S., et al. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19099-4_5
Rui, S., Qiao, Z., Wang, Y., Xu, S., Wang, Y., Wang, L., & Shi, Y. (2026) · IEEE Sensors Journal · doi:10.1109/jsen.2026.3691501
“While LLMs have achieved breakthroughs in various fields due to their powerful representation learning capabilities [26], [27].”
Ye, M., Han, D., Chang, C., Tang, M., Chen, W., & Feng, X. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3694119 · Cited by 1 · see citing works
“For instance, Wang et al. [40] proposed DRG-LLaMA for automatically predicting DRG codes from clinical discharge summaries.”
Expert Consensus on LLM Safety in Clinical Decision Support
Gayathri, K. V., Kanshiya, S., Saranraj, P., Lokeshwari, B., Vilmasen, P., & Wins, J. A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7862-6.ch011
Sultanov, R., Aarthi, S., Srinivasan, S., Ravikumar, R. N., & Umarov, D. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7862-6.ch013
AI-Mediated Access to Cybersecurity Risk Information in Corporate Disclosures
Du, H. W., Li, X., & Du, L. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10774-8
Generative small language models in clinical NLP: applications, adaptation, and evaluation
Türkmen, H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.4000
Ji, D., Huang, M., & Wang, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1913896
Hadef, M., Boulahia, S. Y., & Amamra, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35387-0_45
Organização e Arquitetura da Informação em Large Language Models
Procópio, D. C., & Silva, P. N. (2026) · Informação & Informação · doi:10.5433/1981-8920.2026v31n1p149
Artificial intelligence in laboratory medicine: From machine learning to large language models
Zhang, S., Zhan, Y., Xu, K., & Zhou, H. (2026) · Chinese Medical Journal · doi:10.1097/cm9.0000000000004291
- Authors
- Wang, H., Gao, C., Dantona, C., Hull, B., & Sun, J.
- Affiliations
- Mayo Clinic (US)
- University of Illinois Urbana-Champaign (US)
- Mayo Clinic in Arizona (US)
- Corresponding author
- Sun, Jimeng
- Venue
- npj Digital Medicine
- Published
- 2024-01-22
- Volume
- 7
- Issue
- 1
- Pages
- 16-16
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC10803802
- OpenAlex
- W4391098193
- Semantic Scholar
- c5fa173e6f52dbc2e5156e2e7c9e6d3c0f1c4aa0
- Cited by
- 99
- References
- 23
- Citations by year
- 2026: 22 · 2025: 52 · 2024: 24 · 2023: 1
- FWCI
- 14.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (IIS-2034479, SCH-2014438)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLaMA fine-tuning, low-rank adaptation, clinical notes, discharge summaries, MIMIC-IV, model parameter size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
