CoChat
Sign Up
Cited by 114Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

Large language models streamline automated machine learning for clinical studies

Open access
Tayebi Arasteh, Soroosh, Han, Tianyu, Lotfinia, Mahshad, Kuhl, Christiane, Kather, Jakob Nikolas, Truhn, Daniel, Nebelung, Sven·2024·Nature Communications
DOI 10.1038/s41467-024-45879-8PMC PMC10881983
Open this paper in CoChat — free accountOpen PDF

Abstract

A knowledge gap persists between machine learning (ML) developers (e.g., data scientists) and practitioners (e.g., clinicians), hampering the full utilization of ML for clinical data analysis. We investigated the potential of the ChatGPT Advanced Data Analysis (ADA), an extension of GPT-4, to bridge this gap and perform ML analyses efficiently. Real-world clinical datasets and study details from large trials across various medical specialties were presented to ChatGPT ADA without specific guidance. ChatGPT ADA autonomously developed state-of-the-art ML models based on the original study’s training data to predict clinical outcomes such as cancer development, cancer progression, disease complications, or biomarkers such as pathogenic gene sequences. Following the re-implementation and optimization of the published models, the head-to-head comparison of the ChatGPT ADA-crafted ML models and their respective manually crafted counterparts revealed no significant differences in traditional performance metrics ( p ≥ 0.072). Strikingly, the ChatGPT ADA-crafted ML models often outperformed their counterparts. In conclusion, ChatGPT ADA offers a promising avenue to democratize ML in medicine by simplifying complex data analyses, yet should enhance, not replace, specialized training and resources, to promote broader applications in medical research and practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-024-45879-8

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Study design. Real-world datasets and study details from four large clinical trials were collected and input into the ChatGPT Advanced Data Analysis (ADA) tool. The tool autonomously selected the appropriate machine-learning models for the analysis following prompting. The models were expert-checked and comprehensively evaluated. The ChatGPT ADA-based predictions were compared to the original studies (benchmark publication) and the validatory predictions following the re-implementation of the models. Figure 1 was provided by a freelancer service (fiverr.com). Copyright rests with the authors. The figure constitutes original material and has not been published before.
Fig. 1 Study design. Real-world datasets and study details from four large clinical trials were collected and input into the ChatGPT Advanced Data Analysis (ADA) tool. The tool autonomously selected the appropriate machine-learning models for the analysis following prompting. The models were expert-checked and comprehensively evaluated. The ChatGPT ADA-based predictions were compared to the original studies (benchmark publication) and the validatory predictions following the re-implementation of the models. Figure 1 was provided by a freelancer service (fiverr.com). Copyright rests with the authors. The figure constitutes original material and has not been published before.CC BY · PubMed Central
Fig. 2 Screenshots of an example interaction with ChatGPT ADA to analyze the endocrinologic oncology dataset. ChatGPT ADA autonomously selects and applies the appropriate ML model for the provided dataset, generating predictions for the test data. The model also displays deeper insights in response to follow-up queries about the reasoning and parameters guiding its choices. Note: The “Show work” option visible in the images allows users to view the intermediary Python code offered by the tool.
Fig. 2 Screenshots of an example interaction with ChatGPT ADA to analyze the endocrinologic oncology dataset. ChatGPT ADA autonomously selects and applies the appropriate ML model for the provided dataset, generating predictions for the test data. The model also displays deeper insights in response to follow-up queries about the reasoning and parameters guiding its choices. Note: The “Show work” option visible in the images allows users to view the intermediary Python code offered by the tool.CC BY · PubMed Central
Fig. 3 Benchmark validatory re-implementation—receiver operating characteristic (ROC) curves of ML models as a function of the clinical-trial dataset. The ROC curves of the ChatGPT ADA-based ML model (blue, solid curve) and the validatory ML model as re-implemented by a seasoned data scientist (red, dotted curve) are shown. The True Positive Rate (sensitivity) is plotted versus the False Positive Rate (1-specificity). The diagonal gray line represents the line of no discrimination. Source data are provided as a Source Data file. Bootstrapping54 with replacements and 1000 redraws on the test sets (number of independent samples: Endocrinologic Oncology dataset, n = 295; Gastrointestinal Oncology dataset, n = 6698, Otolaryngology dataset, n = 569; Cardiology dataset, n = 430) was applied to determine means and measures of statistical spread, i.e., standard deviations and 95% confidence intervals (CI). AUROC area under the receiver operating characteristic curve, ChatGPT advanced data analysis.
Fig. 3 Benchmark validatory re-implementation—receiver operating characteristic (ROC) curves of ML models as a function of the clinical-trial dataset. The ROC curves of the ChatGPT ADA-based ML model (blue, solid curve) and the validatory ML model as re-implemented by a seasoned data scientist (red, dotted curve) are shown. The True Positive Rate (sensitivity) is plotted versus the False Positive Rate (1-specificity). The diagonal gray line represents the line of no discrimination. Source data are provided as a Source Data file. Bootstrapping54 with replacements and 1000 redraws on the test sets (number of independent samples: Endocrinologic Oncology dataset, n = 295; Gastrointestinal Oncology dataset, n = 6698, Otolaryngology dataset, n = 569; Cardiology dataset, n = 430) was applied to determine means and measures of statistical spread, i.e., standard deviations and 95% confidence intervals (CI). AUROC area under the receiver operating characteristic curve, ChatGPT advanced data analysis.CC BY · PubMed Central
Fig. 4 Model explainability through the top 10 predictive features for the ChatGPT ADA-selected machine-learning models. An explainability analysis was performed for each clinical trial including (a) Metastatic Disease [Endocrinologic Oncology], b Oesophageal Cancer [Gastrointestinal Oncology], c Hereditary Hearing Loss [Otolaryngology], and d Cardiac Amyloidosis [Cardiology], and ChatGPT ADA-selected machine-learning model. Indicated are SHapley Additive exPlanations (SHAP) values of each predictive feature that measure the feature’s influence on model predictions. High absolute SHAP values signify substantial influence. The features are ranked from top to bottom based on the mean absolute SHAP values (color-coded on the right). In c, specific gene locations are indicated. Please refer to the Methods for more details on abbreviations. Box plots indicate the ranges (x-axes) of each feature (y-axes). Crosses indicate (arithmetic) means, boxes the ranges (first [Q1] to third [Q3] quartile), with the central line representing the (arithmetic) median (second quartile [Q2]). Whiskers extend to 1.5 times the interquartile range above Q3 and below Q1. Any data point outside this range is considered an outlier (dots). Mind the different scales for the color codes and SHAP values. Source data are provided as a Source Data file. ChatGPT ADA performed the SHAP analysis on the training sets (number of independent samples: Endocrinologic Oncology dataset, n = 493, Gastrointestinal Oncology dataset, n = 7899, Otolaryngology dataset, n = 1209, and Cardiology dataset, n = 1712). Plasma MN plasma concentrations of metanephrine, plasma NMN plasma concentrations of normetanephrine, SDHB succinate dehydrogenase complex iron-sulfur subunit B, Plasma MTY plasma concentrations of methoxytyramine, AGC atypical glandular cells, DNA deoxyribonucleic acid, Chron. chronic, Cong. congenital, Dias. diastolic, Sys. systolic. Note: The feature “Hyp. heart w/ HF & Stg 1–4 Unsp. CKD” refers to “Hypertensive heart with heart failure coexisting with unspecified stage 1–4 chronic kidney disease”, while “Prev. hist. PGGLs” refers to “Previous history of Pheochromocytomas and Paragangliomas”.
Fig. 4 Model explainability through the top 10 predictive features for the ChatGPT ADA-selected machine-learning models. An explainability analysis was performed for each clinical trial including (a) Metastatic Disease [Endocrinologic Oncology], b Oesophageal Cancer [Gastrointestinal Oncology], c Hereditary Hearing Loss [Otolaryngology], and d Cardiac Amyloidosis [Cardiology], and ChatGPT ADA-selected machine-learning model. Indicated are SHapley Additive exPlanations (SHAP) values of each predictive feature that measure the feature’s influence on model predictions. High absolute SHAP values signify substantial influence. The features are ranked from top to bottom based on the mean absolute SHAP values (color-coded on the right). In c, specific gene locations are indicated. Please refer to the Methods for more details on abbreviations. Box plots indicate the ranges (x-axes) of each feature (y-axes). Crosses indicate (arithmetic) means, boxes the ranges (first [Q1] to third [Q3] quartile), with the central line representing the (arithmetic) median (second quartile [Q2]). Whiskers extend to 1.5 times the interquartile range above Q3 and below Q1. Any data point outside this range is considered an outlier (dots). Mind the different scales for the color codes and SHAP values. Source data are provided as a Source Data file. ChatGPT ADA performed the SHAP analysis on the training sets (number of independent samples: Endocrinologic Oncology dataset, n = 493, Gastrointestinal Oncology dataset, n = 7899, Otolaryngology dataset, n = 1209, and Cardiology dataset, n = 1712). Plasma MN plasma concentrations of metanephrine, plasma NMN plasma concentrations of normetanephrine, SDHB succinate dehydrogenase complex iron-sulfur subunit B, Plasma MTY plasma concentrations of methoxytyramine, AGC atypical glandular cells, DNA deoxyribonucleic acid, Chron. chronic, Cong. congenital, Dias. diastolic, Sys. systolic. Note: The feature “Hyp. heart w/ HF & Stg 1–4 Unsp. CKD” refers to “Hypertensive heart with heart failure coexisting with unspecified stage 1–4 chronic kidney disease”, while “Prev. hist. PGGLs” refers to “Previous history of Pheochromocytomas and Paragangliomas”.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 53 references

Research works printed but not linked: 2

  1. Ke, G. et al. LightGBM: A highly efficient gradient boosting decision tree. in Advances in Neural Information Processing Systems 30. 3149–3157 (NIPS, 2017).
  2. Lundberg, S. & Lee, S.-I. A Unified Approach to Interpreting Model Predictions. in Proceedings of the 31st International Conference on Neural Information Processing Systems. 4768–4777. https://doi.org/10.5555/3295222.3295230 (NIPS, 2017). · DOI

Cited by

Sorted by citations

Items 101–114 of 114

Related

See all

Metadata

Authors
Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J. N., Truhn, D., & Nebelung, S.
Affiliations
  • Universitätsklinikum Aachen (DE)
  • University Hospital Heidelberg (DE)
Corresponding author
Han, Tianyu
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2024-02-21
Volume
15
Issue
1
Pages
1603-1603
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-024-45879-8
PMC
PMC10881983
OpenAlex
W4392016947
Cited by
114
References
53
Citations by year
2026: 34 · 2025: 53 · 2024: 27
FWCI
10.45
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • RWTH Publications (RWTH Aachen)
Funders
  • European Commission (101057091)
  • Deutsche Forschungsgemeinschaft (TR 1700/7‐1, NE2136/3-1, 2136/3-1)
  • Bundesministerium für Bildung und Forschung (01KD2215B, 031L0312A, 01KX2021)
  • HORIZON EUROPE Framework Programme (101057091)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, ChatGPT, automated machine learning, clinical trials, medical data analysis, GPT-4, clinical outcome prediction, biomarker prediction, tumor progression, model performance comparison
MeSH terms
Machine Learning, Algorithms, Humans, Language, Neoplasms, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review