Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index; 1 of the 6 sources its claims cite were checked for support.

Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information

Open access

Abstract

Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited. Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute’s (NCI’s) answers by using the questions on the “Common Cancer Myths and Misconceptions” web page. The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08). There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.

Overview

CoChat Summary

TL;DR

This brief report evaluated whether ChatGPT provides accurate answers to 13 common cancer myths and misconceptions compared with National Cancer Institute (NCI) answers. Expert reviewers found ChatGPT outputs largely accurate and similar to NCI answers, though with some hedging/uncertainty noted.

Key findings

  • NCI answers were rated accurate by 5 of 5 reviewers for 13 of 13 questions, giving 100% interrater agreement.
  • ChatGPT outputs were rated accurate by 5 of 5 reviewers for 11 of 13 questions, with 96.9% overall agreement; the remaining 2 questions were rated accurate by 4 of 5 reviewers (Fleiss ĸ = ‒0.03, SE = 0.08, P = .6).
  • There were few noticeable differences in word count or Flesch-Kincaid readability between NCI and ChatGPT answers, though both were written at a less readable level than health literacy advocates would likely recommend.
  • Five repeated runs for each of the 13 questions showed subtle sentence-structure changes but consistent answers, suggesting accuracy remained stable with repetitive questioning.

Methods

  • On December 20, 2022, the authors asked ChatGPT (December 15, 2022, version) the 13 questions from the NCI “Common Cancer Myths and Misconceptions” web page.
  • NCI and ChatGPT answers were blinded and independently rated for accuracy (yes vs no) by 5 scientific reviewers with expertise in cancer treatment and cancer misinformation between December 20, 2022, and January 3, 2023.
  • Interrater agreement was assessed with Fleiss ĸ; word count and Flesch-Kincaid grade level were evaluated for each response, and analyses used Stata version 17.

Limitations

  • The paper states that it evaluated only common cancer misinformation in English, so more obscure myths may have incomplete or English-only information and may not be answered accurately.
  • The paper states that ChatGPT outputs may be trained on models that could be scientifically outdated, and ChatGPT was limited to data collected before 2021, so accuracy may be delayed as new scientific information emerges.
  • Evident from the text: this was a small-scale study of 13 questions using one ChatGPT version at one time point; the authors did not code for specific message features, though multiple observers perceived hedging and uncertainty in ChatGPT messages.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jncics/pkad015

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 0 Not valid · 5 Not checked · 17 Own results and interpretations

3 more claims

Create a free account to see all 23 claims and to verify the 5 references still open.

See all claims — free account
  • Probably not a substantial claim. 10
  • It may state two separate claims as one. 5
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 6 references

Cited by

See all 289

Metadata

Authors
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L.
Affiliations
  • University of Utah (US)
  • Yale University (US)
  • Brigham and Women's Hospital (US)
  • University of Florida Health (US)
Corresponding author
Johnson, Skyler B
Venue
JNCI Cancer Spectrum
Published
2023-03-01
Volume
7
Issue
2
ISSN
2515-5091 (ISSN-L)
OpenAlex
W4327681325
Cited by
289
References
6
Citations by year
2026: 21 · 2025: 77 · 2024: 116 · 2023: 75
FWCI
11.54
Type
journal article
Language
en
Open access
Locations
  • JNCI Cancer Spectrum (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Huntsman Cancer Institute
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Social Sciences › Health › Social Media in Health Education
Keywords
ChatGPT, cancer myths, cancer misconceptions, artificial intelligence, chatbots, health information accuracy, National Cancer Institute, information quality, readability
MeSH terms
Artificial Intelligence, Humans, Neoplasms, United States, National Cancer Institute (U.S.)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. https://app.cochat.ai/papers/10.1093/jncics/pkad015
The paper
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023). Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectrum. https://doi.org/10.1093/jncics/pkad015

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.