CoChat

AI-based automation of enrollment criteria and endpoint assessment in clinical trials in liver diseases

Open access

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.

Abstract

Clinical trials in metabolic dysfunction-associated steatohepatitis (MASH, formerly known as nonalcoholic steatohepatitis) require histologic scoring for assessment of inclusion criteria and endpoints. However, variability in interpretation has impacted clinical trial outcomes. We developed an artificial intelligence-based measurement (AIM) tool for scoring MASH histology (AIM-MASH). AIM-MASH predictions for MASH Clinical Research Network necroinflammation grades and fibrosis stages were reproducible ( κ = 1) and aligned with expert pathologist consensus scores ( κ = 0.62–0.74). The AIM-MASH versus consensus agreements were comparable to average pathologists for MASH Clinical Research Network scores (82% versus 81%) and fibrosis (97% versus 96%). Continuous scores produced by AIM-MASH for key histological features of MASH correlated with mean pathologist scores and noninvasive biomarkers and strongly predicted progression-free survival in patients with stage 3 ( P < 0.0001) and stage 4 ( P = 0.03) fibrosis. In a retrospective analysis of the ATLAS trial (NCT03449446), responders receiving study treatment showed a greater continuous change in fibrosis compared with placebo ( P = 0.02). Overall, these results suggest that AIM-MASH may assist pathologists in histologic review of MASH clinical trials, reducing inter-rater variability on trial outcomes and offering a more sensitive and reproducible measure of patient responses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03172-7

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Pipeline for model deployment. a, Input: separate CNN-based models trained with digitized H&E- and MT-stained images annotated by expert pathologists are deployed on H&E- or MT-stained WSIs, respectively, to identify histological features. b, Artifact detection and exclusion: an artifact model, also based on CNNs, detects image and tissue artifacts for both H&E and MT WSIs and excludes them before downstream analysis. c, Image segmentation: H&E and MT CNNs segment and generate pixel-level predictions of relevant histologic features. d, AI-based MASH CRN scoring: CNN pixel-level predictions for each histological feature (for example, fibrosis or steatosis) were clustered using GNN models and a score predicted based on the spatial organization of the cluster. To correct for pathologists’ bias, the GNN models were specified as ‘mixed effects’ models, biases were learned and the GNNs were deployed with predictions using only the unbiased estimate. GNN nodes and edges were built from CNN predictions of relevant histologic features derived from deployment of the H&E, MT and artifact models. e, Output: this two-stage ML approach produced patient-level predictions of MASH CRN MAS component scores and fibrosis stage.
Fig. 1 Pipeline for model deployment. a, Input: separate CNN-based models trained with digitized H&E- and MT-stained images annotated by expert pathologists are deployed on H&E- or MT-stained WSIs, respectively, to identify histological features. b, Artifact detection and exclusion: an artifact model, also based on CNNs, detects image and tissue artifacts for both H&E and MT WSIs and excludes them before downstream analysis. c, Image segmentation: H&E and MT CNNs segment and generate pixel-level predictions of relevant histologic features. d, AI-based MASH CRN scoring: CNN pixel-level predictions for each histological feature (for example, fibrosis or steatosis) were clustered using GNN models and a score predicted based on the spatial organization of the cluster. To correct for pathologists’ bias, the GNN models were specified as ‘mixed effects’ models, biases were learned and the GNNs were deployed with predictions using only the unbiased estimate. GNN nodes and edges were built from CNN predictions of relevant histologic features derived from deployment of the H&E, MT and artifact models. e, Output: this two-stage ML approach produced patient-level predictions of MASH CRN MAS component scores and fibrosis stage.CC BY · PubMed Central
Fig. 2 AI-based detection and scoring of MAS components and fibrosis. The MASH algorithm can detect histopathologic features on WSIs across a range of MASH disease severity. a, Representative H&E-stained slides show AI overlays highlighting regions of steatosis, lobular inflammation and ballooning. Representative cases corresponding to MAS < 4 (total n = 148) and MAS ≥ 4 (total n = 483), according to both pathologist consensus scoring and AI in the test set, are shown. The inset is a magnified field showing the presence of the three MAS components. Scale bar, 0.2 mm. b, Representative MT-stained slides of each MASH CRN fibrosis stage show AI-generated overlays highlighting regions of fibrosis present on biopsies. Representative cases corresponding to MASH CRN fibrosis stages F1 (total n = 159), F2 (total n = 146), F3 (total n = 278) and F4 (total n = 23), according to both pathologist consensus scoring and AI in the test set, are shown. These AI-generated overlays allow for qualitative review of model performance. Scale bar, 0.5 mm.
Fig. 2 AI-based detection and scoring of MAS components and fibrosis. The MASH algorithm can detect histopathologic features on WSIs across a range of MASH disease severity. a, Representative H&E-stained slides show AI overlays highlighting regions of steatosis, lobular inflammation and ballooning. Representative cases corresponding to MAS < 4 (total n = 148) and MAS ≥ 4 (total n = 483), according to both pathologist consensus scoring and AI in the test set, are shown. The inset is a magnified field showing the presence of the three MAS components. Scale bar, 0.2 mm. b, Representative MT-stained slides of each MASH CRN fibrosis stage show AI-generated overlays highlighting regions of fibrosis present on biopsies. Representative cases corresponding to MASH CRN fibrosis stages F1 (total n = 159), F2 (total n = 146), F3 (total n = 278) and F4 (total n = 23), according to both pathologist consensus scoring and AI in the test set, are shown. These AI-generated overlays allow for qualitative review of model performance. Scale bar, 0.5 mm.CC BY · PubMed Central
Fig. 3 AI-based grading/staging of enrollment criteria and efficacy endpoints. a, Model-derived scores distinguished fibrosis stages F1–F3 versus F4 and MAS ≥ 4 (with each component grade ≥1) versus MAS < 4, criteria used to determine trial enrollment, using biopsies from the STELLAR-3 and STELLAR-4 clinical trials (n = 605). AIM-MASH agreement with consensus was comparable to that of each pathologist. Bar plots represent the point estimate of each enrollment criteria endpoint, and whiskers represent the 95% CIs estimated using 10,000 bootstrap samples. b, For assessment of efficacy endpoints commonly used in phase 2b and phase 3 MASH clinical trials, AIM-MASH agreement with consensus was comparable to that of an average pathologist. Assessment was performed on an external held-out validation dataset from a phase 2b MASH clinical trial using biopsies of patients meeting the following endpoints: fibrosis improvement without MASH worsening (n = 279), MASH resolution without fibrosis worsening (n = 279) and MAS reduction ≥2 (n = 326). Bar plots represent the point estimate of each enrollment criteria endpoint, and whiskers represent the 95% CIs estimated using 10,000 bootstrap samples.
Fig. 3 AI-based grading/staging of enrollment criteria and efficacy endpoints. a, Model-derived scores distinguished fibrosis stages F1–F3 versus F4 and MAS ≥ 4 (with each component grade ≥1) versus MAS < 4, criteria used to determine trial enrollment, using biopsies from the STELLAR-3 and STELLAR-4 clinical trials (n = 605). AIM-MASH agreement with consensus was comparable to that of each pathologist. Bar plots represent the point estimate of each enrollment criteria endpoint, and whiskers represent the 95% CIs estimated using 10,000 bootstrap samples. b, For assessment of efficacy endpoints commonly used in phase 2b and phase 3 MASH clinical trials, AIM-MASH agreement with consensus was comparable to that of an average pathologist. Assessment was performed on an external held-out validation dataset from a phase 2b MASH clinical trial using biopsies of patients meeting the following endpoints: fibrosis improvement without MASH worsening (n = 279), MASH resolution without fibrosis worsening (n = 279) and MAS reduction ≥2 (n = 326). Bar plots represent the point estimate of each enrollment criteria endpoint, and whiskers represent the 95% CIs estimated using 10,000 bootstrap samples.CC BY · PubMed Central
Fig. 4 AIM-based retrospective drug efficacy assessment. AIM-MASH models were deployed on WSIs from baseline and week 48 biopsies from patients enrolled in the phase 2b ATLAS trial, which evaluated combination therapies for individuals with advanced MASH fibrosis. a, For the trial endpoints of MAS ≥ 2-point improvement, fibrosis improvement without worsening of MASH and MASH resolution without worsening of fibrosis, AIM-MASH models showed a greater proportion of responders compared with that determined by the trial central reader. For MAS ≥ 2-point improvement, odd ratios (ORs) for AI and central reader were 5.1 (95% CI 2.0–13.1) and 5.7 (95% CI 1.6–20.2), respectively; Cochran–Mantel–Haenszel (CMH) test statistics were 11.9 (P = 0.0006) and 7.9 (P = 0.005), respectively. For fibrosis improvement without worsening of MASH, ORs for AI and central reader were 2.2 (95% CI 0.7–6.3) and 2.2 (95% CI 0.6–7.7), respectively; CMH test statistics were 2.1 (P = 0.152) and 1.7 (P = 0.196), respectively. For MASH resolution without worsening of fibrosis, OR for AI was 2.7 (95% CI 0.8–8.8); OR for central reader was undefined, as no placebo responders were identified. CMH test statistics were 2.7 (P = 0.101) for AI and 2.0 (P = 0.155) for central reader. Sample sizes varied depending on data availability. b, The placebo-adjusted response rate detected by AIM-MASH was greater than that detected by the central reader.
Fig. 4 AIM-based retrospective drug efficacy assessment. AIM-MASH models were deployed on WSIs from baseline and week 48 biopsies from patients enrolled in the phase 2b ATLAS trial, which evaluated combination therapies for individuals with advanced MASH fibrosis. a, For the trial endpoints of MAS ≥ 2-point improvement, fibrosis improvement without worsening of MASH and MASH resolution without worsening of fibrosis, AIM-MASH models showed a greater proportion of responders compared with that determined by the trial central reader. For MAS ≥ 2-point improvement, odd ratios (ORs) for AI and central reader were 5.1 (95% CI 2.0–13.1) and 5.7 (95% CI 1.6–20.2), respectively; Cochran–Mantel–Haenszel (CMH) test statistics were 11.9 (P = 0.0006) and 7.9 (P = 0.005), respectively. For fibrosis improvement without worsening of MASH, ORs for AI and central reader were 2.2 (95% CI 0.7–6.3) and 2.2 (95% CI 0.6–7.7), respectively; CMH test statistics were 2.1 (P = 0.152) and 1.7 (P = 0.196), respectively. For MASH resolution without worsening of fibrosis, OR for AI was 2.7 (95% CI 0.8–8.8); OR for central reader was undefined, as no placebo responders were identified. CMH test statistics were 2.7 (P = 0.101) for AI and 2.0 (P = 0.155) for central reader. Sample sizes varied depending on data availability. b, The placebo-adjusted response rate detected by AIM-MASH was greater than that detected by the central reader.CC BY · PubMed Central
Fig. 5 AI-based continuous MASH CRN scores. a, Correlation of AI-based continuous scores with mean scores across three pathologists from EMMINENCE in the analytic performance test set. Results are shown for both AI-derived ordinal bins (blue) and pathologist-derived ordinal bins (gray). Plotted values were derived from Kendall’s tau (τ) rank correlation analysis. FDR correction of P values was performed using the Benjamini–Hochberg procedure. Filled circles indicate statistical significance, FDR-corrected P < 0.05. b, cFib versus CPA measurements in primary endpoint responders in the ATLAS clinical trial. cFib and CPA were compared between patients receiving treatment and placebo using two-sided Mann–Whitney U tests. In primary endpoint responders, continuous fibrosis scores were significantly reduced in treated patients (n = 17) versus placebo patients (n = 6; Mann–Whitney U = 20.0, P = 0.02), while proportionate area fibrosis measurements were not significantly reduced (Mann–Whitney U = 39.0, P = 0.21). cFib and CPA values for patients classified as nonresponders (n = 76), in the treatment (n = 45) or placebo (n = 31) group, are also shown. Boxes represent the 25th percentile, median and 75th percentile of the data. Whiskers extend to points that lie within 1.5-fold of the inter-quartile range of the 25th and 75th percentiles. c, Stratification of patients with BL F3 or F4 fibrosis from STELLAR-3 and STELLAR-4 trial cohorts into rapid (red) and slow (orange) progressors based on continuous score cutoffs of 3.6 and 4.6, respectively. Kaplan–Meier and Cox proportional hazards regression analyses are shown. F3: log-rank statistic = 31.0, P = 2.6 × 10−8; F4: log-rank statistic = 4.8, P = 0.028. Rounded cutoffs were chosen to maximize hazards. d, Discriminatory accuracy of AI-derived continuous scores versus ordinal scores to predict progression to cirrhosis (left) and LRE (right) in STELLAR-3 and STELLAR-4 trial cohorts. In both cases, using receiver operating characteristic analysis, the continuous AUC was significantly greater (progression to cirrhosis: 0.66 (95% CI 0.60–0.71) versus 0.59 (95% CI 0.55–0.60); progression to LRE: 0.61 (95% CI 0.51–0.71) versus 0.54 (95% CI 0.47–59)). AUC, area under the receiver operating characteristic curve; BL, baseline; FDR, false discovery rate; FPR, false positive rate; τ, Kendall’s rank correlation coefficient for ordinal scores; TPR, true positive rate.
Fig. 5 AI-based continuous MASH CRN scores. a, Correlation of AI-based continuous scores with mean scores across three pathologists from EMMINENCE in the analytic performance test set. Results are shown for both AI-derived ordinal bins (blue) and pathologist-derived ordinal bins (gray). Plotted values were derived from Kendall’s tau (τ) rank correlation analysis. FDR correction of P values was performed using the Benjamini–Hochberg procedure. Filled circles indicate statistical significance, FDR-corrected P < 0.05. b, cFib versus CPA measurements in primary endpoint responders in the ATLAS clinical trial. cFib and CPA were compared between patients receiving treatment and placebo using two-sided Mann–Whitney U tests. In primary endpoint responders, continuous fibrosis scores were significantly reduced in treated patients (n = 17) versus placebo patients (n = 6; Mann–Whitney U = 20.0, P = 0.02), while proportionate area fibrosis measurements were not significantly reduced (Mann–Whitney U = 39.0, P = 0.21). cFib and CPA values for patients classified as nonresponders (n = 76), in the treatment (n = 45) or placebo (n = 31) group, are also shown. Boxes represent the 25th percentile, median and 75th percentile of the data. Whiskers extend to points that lie within 1.5-fold of the inter-quartile range of the 25th and 75th percentiles. c, Stratification of patients with BL F3 or F4 fibrosis from STELLAR-3 and STELLAR-4 trial cohorts into rapid (red) and slow (orange) progressors based on continuous score cutoffs of 3.6 and 4.6, respectively. Kaplan–Meier and Cox proportional hazards regression analyses are shown. F3: log-rank statistic = 31.0, P = 2.6 × 10−8; F4: log-rank statistic = 4.8, P = 0.028. Rounded cutoffs were chosen to maximize hazards. d, Discriminatory accuracy of AI-derived continuous scores versus ordinal scores to predict progression to cirrhosis (left) and LRE (right) in STELLAR-3 and STELLAR-4 trial cohorts. In both cases, using receiver operating characteristic analysis, the continuous AUC was significantly greater (progression to cirrhosis: 0.66 (95% CI 0.60–0.71) versus 0.59 (95% CI 0.55–0.60); progression to LRE: 0.61 (95% CI 0.51–0.71) versus 0.54 (95% CI 0.47–59)). AUC, area under the receiver operating characteristic curve; BL, baseline; FDR, false discovery rate; FPR, false positive rate; τ, Kendall’s rank correlation coefficient for ordinal scores; TPR, true positive rate.CC BY · PubMed Central
Extended Data Fig. 1 AIM-MASH CNN and GNN model training and predictions. CNN, convolutional neural network; GNN, graph neural network; AIM, Intelligence based Measurement; MASH, metabolic dysfunction-associated steatohepatitis.
Extended Data Fig. 1 AIM-MASH CNN and GNN model training and predictions. CNN, convolutional neural network; GNN, graph neural network; AIM, Intelligence based Measurement; MASH, metabolic dysfunction-associated steatohepatitis.CC BY · PubMed Central
Extended Data Fig. 2 AIM-MASH H&E and Trichrome inference pipelines. H&E, hematoxylin and eosin; TC, trichrome.
Extended Data Fig. 2 AIM-MASH H&E and Trichrome inference pipelines. H&E, hematoxylin and eosin; TC, trichrome.CC BY · PubMed Central
Extended Data Fig. 3 Segmentation model development process. QC, quality control.
Extended Data Fig. 3 Segmentation model development process. QC, quality control.CC BY · PubMed Central
Extended Data Fig. 4 Mapping of continuous scores. Continuous MAS (steatosis, ballooning and lobular inflammation) and CRN fibrosis scores were produced by mapping GNN-derived ordinal grades/stages to bins, such that ordinal scores are spread over a continuous range (unit distance of 1). This continuous scoring system allows a more granular measurement of histological changes occurring at the subordinal level, while maintaining fidelity to the accepted ordinal scoring system.
Extended Data Fig. 4 Mapping of continuous scores. Continuous MAS (steatosis, ballooning and lobular inflammation) and CRN fibrosis scores were produced by mapping GNN-derived ordinal grades/stages to bins, such that ordinal scores are spread over a continuous range (unit distance of 1). This continuous scoring system allows a more granular measurement of histological changes occurring at the subordinal level, while maintaining fidelity to the accepted ordinal scoring system.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 5

  1. Pedregosa, F., Bach, F. & Gramfort, A. On the consistency of ordinal regression methods. J. Mach. Learn. Res. 18, 1–35 (2017).
  2. Iyer, J. S. et al. Artificial intelligence-based measurement of NASH histology (AIM-NASH) recapitulates primary results from phase 3 study of resmetirom for treatment of NASH/MASH. Hepatology 79, E56–E57 (2023).
  3. Cable, E. E., McWherter, C. A., Choi, Y. J., Shah, S. & Carrasco-Zevallos, O. M. Quantitative assessment of NASH pathologies in 152 baseline H&E slides from a phase 2 trial using the PathAI machine learning algorithm. NASH-TAG Proceedings 5, 5 (2021).
  4. Shevell, D. E. et al. Comparison of manual vs. machine learning approaches to liver biopsy scoring for NASH and fibrosis: a post-hoc analysis of the FALCON 1 study. Hepatology 74, 1415A (2021).
  5. Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. in Advances In Neural Information Processing Systems (eds Pereira, F., et al.) vol. 25 (Curran Associates, 2012).

Cited by

See all 66

Metadata

Authors
Iyer, J. S., Juyal, D., Le, Q., Shanis, Z., Pokkalla, H., Pouryahya, M., Pedawi, A., Stanford-Moore, S. A., Biddle-Snead, C., Carrasco-Zevallos, O., Lin, M., Egger, R., Hoffman, S., Elliott, H., Leidal, K., Myers, R. P., Chung, C., Billin, A. N., Watkins, T. R., Patterson, S. D., Resnick, M., Wack, K., Glickman, J., Burt, A. D., Loomba, R., Sanyal, A. J., Glass, B., Montalto, M. C., Taylor-Weiner, A., Wapinski, I., & Beck, A. H.
Affiliations
  • PathAI (United States) (US)
  • Harvard University (US)
  • Gilead Sciences (United States) (US)
  • Rhode Island Hospital (US)
  • Newcastle University (GB)
  • University of California San Diego (US)
  • Virginia Commonwealth University (US)
  • Amgen (United States) (US)
Corresponding author
Beck, Andrew H.
Venue
Nature Medicine
Published
2024-08-07
Volume
30
Issue
10
Pages
2914-2923
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4401397580
Cited by
54
References
45
Citations by year
2026: 23 · 2025: 25 · 2024: 6
FWCI
16.48
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • Newcastle University ePrints (Newcastle Univesity) (cc-by)
Funders
  • National Institutes of Health (UL1TR001442)
  • National Institute on Alcohol Abuse and Alcoholism (U01AA029019)
  • National Heart, Lung, and Blood Institute (P01HL147835)
  • National Institute of Diabetes and Digestive and Kidney Diseases (R01DK124318, R01DK121378, R01DK106419, P30DK120515, U01DK061734, U01DK130190)
  • National Center for Advancing Translational Sciences
Topics
  • Health Sciences › Medicine › Hepatology › Liver Disease Diagnosis and Treatment
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
artificial intelligence, histologic scoring, nonalcoholic steatohepatitis, liver fibrosis, clinical trial endpoints, interobserver variability, ATLAS trial, progression-free survival, liver disease
MeSH terms
Artificial Intelligence, Automation, Clinical Trials as Topic, Female, Humans, Liver Cirrhosis, Liver Diseases, Male, Retrospective Studies, Reproducibility of Results, Patient Selection, Endpoint Determination, Non-alcoholic Fatty Liver Disease

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-based automation of enrollment criteria and endpoint assessment in clinical trials in liver diseases. https://app.cochat.ai/papers/10.1038/s41591-024-03172-7
The paper
Iyer, J. S., Juyal, D., Le, Q., Shanis, Z., Pokkalla, H., Pouryahya, M., Pedawi, A., Stanford-Moore, S. A., Biddle-Snead, C., Carrasco-Zevallos, O., Lin, M., Egger, R., Hoffman, S., Elliott, H., Leidal, K., Myers, R. P., Chung, C., Billin, A. N., Watkins, T. R., . . . Beck, A. H. (2024). AI-based automation of enrollment criteria and endpoint assessment in clinical trials in liver diseases. Nature Medicine. https://doi.org/10.1038/s41591-024-03172-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.