Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma

Open access

Preprint — published as 10.3350/cmh.2023.0089 · Paper page

Abstract

Background: Patients with cirrhosis and hepatocellular carcinoma (HCC) require extensive and personalized care to improve outcomes. ChatGPT (Generative Pre-trained Transformer), a natural language processing model, holds potential to provide professional yet patient-friendly support. Aim: Examining the accuracy and reproducibility of ChatGPT in answering questions regarding knowledge, management, and emotional support for cirrhosis and HCC. Method: ChatGPT’s responses to 164 questions were independently graded by two transplant hepatologists and resolved by a third reviewer. The performance of ChatGPT was compared to physicians or trainees in two validated questionnaires. ChatGPT’s knowledge on cirrhosis care was tested using 26 quality measures of cirrhosis management. Finally, its emotional support capacity was tested. Results: ChatGPT regurgitated extensive knowledge of cirrhosis and HCC, but for questions with correct responses, only a small proportion was labelled as comprehensive. The performance was better in basic knowledge, lifestyle, and treatment than in the domains of diagnosis and preventive medicine. For the quality measures, the model answered 76.9% of questions correctly but failed to specify decision-making cut-offs and treatment durations. Compared to physicians/trainees, ChatGPT lacked knowledge of regional guidelines variations, such as HCC screening criteria. However, it provided practical and multifaceted advice to patients and caregivers regarding the next steps and adjusting to a new diagnosis. Conclusion: In summary, we analyzed the areas of robustness and limitations of ChatGPT’s responses on the management of cirrhosis and HCC and relevant emotional support. ChatGPT may have a role as an adjunct informational tool for patients and physicians to improve outcomes. Conflict of Interest Disclosures: None declared. Funding/Support: None Ethics Approval: Since all responses from ChatGPT were publicly available, approval from the institutional review board was not sought. Guarantors of article: Dr. Kuo and Spiegel Author Contributions: Concept and design: Yeo, Samaan, Spiegel, Kuo Acquisition of data: Yeo, Samaan, Ng, Vipani Data review: Ting, Trivedi, Kuo Statistical analysis: Yeo Drafting of the manuscript: Yeo, Samaan, Ng Critical revision of the manuscript: All authors

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.02.06.23285449

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Flow chart of question selection for cirrhosis and hepatocellular carcinoma (HCC). Frequently asked questions about the knowledge and management of cirrhosis or HCC were collected from patient support groups on Facebook and well-regarded professional societies and institutions. FAQs, frequently asked questions.
Figure 1. Flow chart of question selection for cirrhosis and hepatocellular carcinoma (HCC). Frequently asked questions about the knowledge and management of cirrhosis or HCC were collected from patient support groups on Facebook and well-regarded professional societies and institutions. FAQs, frequently asked questions.CC BY-NC · PubMed Central
Figure 2. Grade of responses by the ChatGPT language model to questions related to (A) cirrhosis and (B) hepatocellular carcinoma (HCC). The percentage of responses being graded as comprehensive, correct but inadequate, mixed with correct and incorrect/outdated data, and completely incorrect were provided. GPT, Generative Pre-trained Transformer.
Figure 2. Grade of responses by the ChatGPT language model to questions related to (A) cirrhosis and (B) hepatocellular carcinoma (HCC). The percentage of responses being graded as comprehensive, correct but inadequate, mixed with correct and incorrect/outdated data, and completely incorrect were provided. GPT, Generative Pre-trained Transformer.CC BY-NC · PubMed Central
Image 3
CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Cited by

See all 687

Metadata

Authors
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., & Kuo, A.
Affiliations
  • Cedars-Sinai Medical Center (US)
  • University of Bristol (GB)
  • Tulane University (US)
Corresponding author
  • Spiegel, Brennan
  • Kuo, Alexander
Venue
Clinical and Molecular Hepatology
Publisher
openRxiv
Published
2023-02-08
Volume
29
Issue
3
Pages
721-732
ISSN
2287-2728 (ISSN-L)
OpenAlex
W4353016766
Cited by
666
References
28
Citations by year
2026: 78 · 2025: 197 · 2024: 259 · 2023: 128
FWCI
26.44
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Clinical and Molecular Hepatology (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
ChatGPT, large language models, liver cirrhosis, hepatocellular carcinoma, patient education, emotional support, HCC screening, cirrhosis management, quality metrics, accuracy assessment, artificial intelligence in healthcare
MeSH terms
Carcinoma, Hepatocellular, Humans, Liver Cirrhosis, Liver Neoplasms, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. https://app.cochat.ai/papers/10.1101/2023.02.06.23285449
The paper
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., & Kuo, A. (2023). Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clinical and Molecular Hepatology. https://doi.org/10.1101/2023.02.06.23285449

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.