The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index; 1 of the 6 sources its claims cite were checked for support.
Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information
Open accessAbstract
Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited. Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute’s (NCI’s) answers by using the questions on the “Common Cancer Myths and Misconceptions” web page. The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08). There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.
Overview
CoChat Summary
TL;DR
This brief report evaluated whether ChatGPT provides accurate answers to 13 common cancer myths and misconceptions compared with National Cancer Institute (NCI) answers. Expert reviewers found ChatGPT outputs largely accurate and similar to NCI answers, though with some hedging/uncertainty noted.
Key findings
- NCI answers were rated accurate by 5 of 5 reviewers for 13 of 13 questions, giving 100% interrater agreement.
- ChatGPT outputs were rated accurate by 5 of 5 reviewers for 11 of 13 questions, with 96.9% overall agreement; the remaining 2 questions were rated accurate by 4 of 5 reviewers (Fleiss ĸ = ‒0.03, SE = 0.08, P = .6).
- There were few noticeable differences in word count or Flesch-Kincaid readability between NCI and ChatGPT answers, though both were written at a less readable level than health literacy advocates would likely recommend.
- Five repeated runs for each of the 13 questions showed subtle sentence-structure changes but consistent answers, suggesting accuracy remained stable with repetitive questioning.
Methods
- On December 20, 2022, the authors asked ChatGPT (December 15, 2022, version) the 13 questions from the NCI “Common Cancer Myths and Misconceptions” web page.
- NCI and ChatGPT answers were blinded and independently rated for accuracy (yes vs no) by 5 scientific reviewers with expertise in cancer treatment and cancer misinformation between December 20, 2022, and January 3, 2023.
- Interrater agreement was assessed with Fleiss ĸ; word count and Flesch-Kincaid grade level were evaluated for each response, and analyses used Stata version 17.
Limitations
- The paper states that it evaluated only common cancer misinformation in English, so more obscure myths may have incomplete or English-only information and may not be answered accurately.
- The paper states that ChatGPT outputs may be trained on models that could be scientifically outdated, and ChatGPT was limited to data collected before 2021, so accuracy may be delayed as new scientific information emerges.
- Evident from the text: this was a small-scale study of 13 questions using one ChatGPT version at one time point; the authors did not code for specific message features, though multiple observers perceived hedging and uncertainty in ChatGPT messages.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 5 Not checked · 17 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT.”
- Also stated in: § Abstract · plain
“There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT.”
§ Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The remaining 2 questions (question 5, “Does my attitude—positive or negative—determine my risk of getting or surviving cancer?” and question 9, “Do power lines cause cancer?”) were rated as accurate by 4 of 5 reviewers, with a single unique reviewer rating each question as inaccurate (ĸ = ‒0.03, SE = 0.08, P = .6).”
§ Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we completed 5 runs repeatedly from the same input for each of the 13 questions (Supplementary Table 1, available online). On review, although subtle sentence structure changes were noted, the answers remained consistent”
§ Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Answers from both sources were written at a less readable rate than would likely be recommended by health literacy advocates, for example.”
§ Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information > Abstract
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that answers remain accurate with repetitive questioning.”
§ Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information > Abstract
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
3 more claims
Create a free account to see all 23 claims and to verify the 5 references still open.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 5
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“it was observed that ChatGPT responses were relevant and had a high level of accuracy, similar to the responses of the National Cancer Institute [32].”
From Is there any room for ChatGPT AI bot in speech-language pathology? (2025) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“ChatGPT was shown to exhibit good performance at answering questions about cancer myths and misconceptions (159).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [159]
Citations
Cites
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Interrater agreement was evaluated by Fleiss Œ (5).”
MethodDissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“There is concern that algorithms reinforce current health disparities and inequities (9), although the extent to which this is true about health information specifically is currently unknown.”
BackgroundChatbots in the fight against the COVID-19 pandemic
Miner, A. S., Laranjo, L., & Kocaballi, A. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0280-0 · Cited by 322
An Initial Model of Trust in Chatbots for Customer Service—Findings from a Questionnaire Study
Nordheim, C. B., Følstad, A., & Bjørkli, C. A. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz022 · Cited by 233
Cancer misinformation and harmful information on Facebook and other social media: a brief report
Johnson, S. B., Parsons, M., Dorff, T., Moran, M. S., Ward, J. H., Cohen, S. A., Akerley, W., Bauman, J., Hubbard, J., Spratt, D. E., et al. (2022) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djab141 · Cited by 210
Self-citationCalixte, R., Rivera, A., Oridota, O., Beauchamp, W., & Camacho‐Rivera, M. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17186856 · Cited by 124
Cited by
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
ChatGPT for healthcare services: An emerging stage for an innovative perspective
Javaid, M., Haleem, A., & Singh, R. P. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100105 · Cited by 429
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
Self-citationChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model
Huang, H., Zheng, O., Wang, D., Yin, J., Wang, Z., Ding, S., Yin, H., Xu, C., Yang, R., Zheng, Q., et al. (2023) · International Journal of Oral Science · doi:10.1038/s41368-023-00239-y · Cited by 324
Metadata
- Authors
- Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L.
- Affiliations
- University of Utah (US)
- Yale University (US)
- Brigham and Women's Hospital (US)
- University of Florida Health (US)
- Corresponding author
- Johnson, Skyler B
- Venue
- JNCI Cancer Spectrum
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-03-01
- Volume
- 7
- Issue
- 2
- ISSN
- 2515-5091 (ISSN-L)
- PMC
- PMC10020140
- OpenAlex
- W4327681325
- Cited by
- 289
- References
- 6
- Citations by year
- 2026: 21 · 2025: 77 · 2024: 116 · 2023: 75
- FWCI
- 11.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JNCI Cancer Spectrum (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Huntsman Cancer Institute
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Social Sciences › Health › Social Media in Health Education
- Keywords
- ChatGPT, cancer myths, cancer misconceptions, artificial intelligence, chatbots, health information accuracy, National Cancer Institute, information quality, readability
- MeSH terms
- Artificial Intelligence, Humans, Neoplasms, United States, National Cancer Institute (U.S.)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. https://app.cochat.ai/papers/10.1093/jncics/pkad015
- The paper
- Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023). Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectrum. https://doi.org/10.1093/jncics/pkad015
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.