CoChat
Sign Up
PreprintCited by 220Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIMisinformation and Its Impacts

Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions

Open access
Gravel, Jocelyn, D’Amours-Gravel, Madeleine, Osmanlliu, Esli·2023·Mayo Clinic Proceedings Digital Health
DOI 10.1101/2023.03.16.23286914PMC PMC11975740

Preprint — published as 10.1016/j.mcpdig.2023.05.004 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: ChatGPT have gained public notoriety and recently supported manuscript preparation. Our objective was to evaluate the quality of the answers and the references provided by ChatGPT for medical questions. Methods: Three researchers asked ChatGPT a total of 20 medical questions and prompted it to provide the corresponding references. The responses were evaluated for quality of content by medical experts using a verbal numeric scale going from 0 to 100%. These experts were the corresponding author of the 20 articles from where the medical questions were derived. We planned to evaluate three references per response for their pertinence, but this was amended based on preliminary results showing that most references provided by ChatGPT were fabricated. Results: ChatGPT provided responses varying between 53 and 244 words long and reported two to seven references per answer. Seventeen of the 20 invited raters provided feedback. The raters reported limited quality of the responses with a median score of 60% (1 st and 3 rd quartile: 50% and 85%). Additionally, they identified major (n=5) and minor (n=7) factual errors among the 17 evaluated responses. Of the 59 references evaluated, 41 (69%) were fabricated, though they appeared real. Most fabricated citations used names of authors with previous relevant publications, a title that seemed pertinent and a credible journal format. Interpretation: When asked multiple medical questions, ChatGPT provided answers of limited quality for scientific publication. More importantly, ChatGPT provided deceptively real references. Users of ChatGPT should pay particular attention to the references provided before integration into medical manuscripts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.03.16.23286914

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The zero-trust framework proposes to use specific generative AI systems designed for educational purposes that expose students to false and misleading information that they must recognize and criticize to complete specifically designed tests.

    “This framework proposes to use specific generative AI systems designed for educational purposes that expose students to false and misleading information that they must recognize and criticize to complete specifically designed tests (Dan et al., 2023;Gravel et al., 2023;Stratachery, 2022).”

    From ChatGPT and the Technology-Education Tension: Applying Contextual Virtue Epistemology to a Cognitive Artifact (2024) · cites this paper as Gravel et al., 2023

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. Kitamura FC. ChatGPT is shaping the future of medical writing but still requires human judgment. Radiology. 2023;307(2): e230171. https://doi.org/10.1148/radiol.230171. · DOI · Paper page
  2. Biswas S. ChatGPT and the future of medical writing. Radiology. 2023;307(2):e223312. https://doi.org/10.1148/radiol.223312. · DOI · Paper page
  3. O'Connor S. Open artificial intelligence platforms in nursing ed- ucation: Tools for academic progress or abuse? Nurse Educ Pract. 2023;66:103537. https://doi.org/10.1016/j.nepr.2022.10 3537. · DOI
  4. Chat GPT Generative pre-trained transformer, Zhavoronkov A. Rapamycin in the context of Pascal's Wager: generative pre-trained transformer perspective. Oncoscience. 2022;9:82-84.
  5. Teixeira da Silva JA. Is ChatGPT a valid author? Nurse Educ Pract. 2023;68:103600. https://doi.org/10.1016/j.nepr.2023.103 600. · DOI
  6. Else H. Abstracts written by ChatGPT fool scientists. Nature. 2023;613(7944):423. https://doi.org/10.1038/d41586-023-00 056-7. · DOI
  7. Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. Preprint. Posted online December 27, 2022, https://www.biorxiv.org/content/10.1101/2022.12. 23.521610v1. Accessed February 7, 2023. · DOI
  8. Cahan P, Treutlein B. A conversation with ChatGPT on the role of computational systems biology in stem cell research. Stem Cell Rep. 2023;18(1):1-2. https://doi.org/10.1016/j.stemcr. 2022.12.009. · DOI · Paper page
  9. Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care. 2023;27(1):75. https://doi. org/10.1186/s13054-023-04380-2. · DOI · Paper page
  10. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11(6). https://doi. org/10.3390/healthcare11060887. · DOI · Paper page
  11. Thorp HH. ChatGPT is fun, but not an author. Science. 2023; 379(6630):313. https://doi.org/10.1126/science.adg7879. · DOI · Paper page
  12. Stokel-Walker C. ChatGPT listed as author on research papers: many scientists disapprove. Nature. 2023;613(7945):620-621. https://doi.org/10.1038/d41586-023-00107-z. · DOI
  13. Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature. 2023;613(7945):612. https://doi.org/10.1038/d41586-023-00191-1. · DOI · Paper page
  14. Looi MK. Sixty seconds on. ChatGPT. BMJ. 2023;380:205. https://doi.org/10.1136/bmj.p205. · DOI
  15. Will ChatGPT transform healthcare? Nat Med. 2023;29(3):505- 506.
  16. Weatherald J, Parhar KKS, Al Duhailib Z, et al. Efficacy of awake prone positioning in patients with covid-19 related hypoxemic respiratory failure: systematic review and meta-analysis of ran- domized trials. BMJ. 2022;379:e071966.
  17. Xie J, Wang M, Long Z, et al. Global burden of type 2 diabetes in adolescents and young adults, 1990-2019: systematic analysis of the Global Burden of Disease Study 2019. BMJ. 2022;379: e072385.
  18. Santer M, Muller I, Becque T, et al. Eczema Care Online be havioural interventions to support self-care for children and young people: two independent, pragmatic, randomised controlled trials. BMJ. 2022;379:e072007.
  19. McIlroy DR, Shotwell MS, Lopez MG, et al. Oxygen administrat ion during surgery and postoperative organ injury: observational cohort study. BMJ. 2022;379:e070941.
  20. Candal-Pedreira C, Ross JS, Ruano-Ravina A, Egilman DS, Fernandez E, Perez-Rios M. Retracted papers originating from paper mills: cross sectional study. BMJ. 2022;379:e071517.
  21. Jefferies K, Martin-Misener R, Murphy GT, Gahagan J, Bernard WT. African Nova Scotian nurses' perceptions and ex- periences of leadership: a qualitative study informed by Black feminist theory. CMAJ. 2022;194(42):E1437-E1447. https://doi. org/10.1503/cmaj.220019. · DOI
  22. Zhu Z, Huang JY, Ruan G, et al. Metformin use and associated risk of total joint replacement in patients with type 2 diabetes: a population-based matched cohort study. CMAJ. 2022;194(49): E1672-E1684. https://doi.org/10.1503/cmaj.220952. · DOI
  23. Rudoler D, Peterson S, Stock D, et al. Changes over time in pa- tient visits and continuity of care among graduating cohorts of family physicians in 4 Canadian provinces. CMAJ. 2022;194(48): E1639-E1646. https://doi.org/10.1503/cmaj.220439. · DOI
  24. Skowronski DM, Kaweski SE, Irvine MA, et al. Serial cross- sectional estimation of vaccine-and infection-induced SARS- CoV-2 seroprevalence in British Columbia, Canada. CMAJ. 2022;194(47):E1599-E1609. https://doi.org/10.1503/cmaj.22 1335. · DOI
  25. Naveed Z, Li J, Spencer M, et al. Observed versus expected rates of myocarditis after SARS-CoV-2 vaccination: a population-based cohort study. CMAJ. 2022;194(45):E1529- E1536. https://doi.org/10.1503/cmaj.220676. · DOI
  26. Kalra PR, Cleland JGF, Petrie MC, et al. Intravenous ferric deri- somaltose in patients with heart failure and iron deficiency in the UK (IRONMAN): an investigator-initiated, prospective, randomised, open-label, blinded-endpoint trial. Lancet. 2022; 400(10369):2199-2209. https://doi.org/10.1016/S0140-6736 (22)02083-9. · DOI
  27. Krystal JH, Kane JM, Correll CU, et al. Emraclidine, a novel pos- itive allosteric modulator of cholinergic M4 receptors, for the treatment of schizophrenia: a two-part, randomised, double- blind, placebo-controlled, phase 1b trial. Lancet. 2022; 400(10369):2210-2220. https://doi.org/10.1016/S0140-6736 (22)01990-0. · DOI
  28. Sheikh J, Allotey J, Kew T, et al. Effects of race and ethnicity on perinatal outcomes in high-income and upper-middle-income countries: an individual participant data meta-analysis of 2 198 655 pregnancies. Lancet. 2022;400(10368):2049-2062. https:// doi.org/10.1016/S0140-6736(22)01191-6. · DOI
  29. Kramer CK, Leitao CB, Viana LV. The impact of urbanisation on the cardiometabolic health of Indigenous Brazilian peoples: a systematic review and meta-analysis, and data from the Brazilian health registry. Lancet. 2022;400(10368):2074-2083. https://doi. org/10.1016/S0140-6736(22)00625-0. · DOI
  30. Huh S. Are ChatGPT's knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study. J Educ Eval Health Prof. 2023;20:1. https://doi.org/10.3352/jeehp.2023. 20.1. · DOI
  31. Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9: e45312. https://doi.org/10.2196/45312. · DOI · Paper page
  32. van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature. 2023;614(7947):224-226. https://doi.org/10.1038/d41586- 023-00288-7. · DOI · Paper page
  33. Macdonald C, Adeloye D, Sheikh A, Rudan I. Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. J Glob Health. 2023;13:01003. https://doi. org/10.7189/jogh.13.01003. · DOI · Paper page
  34. Akhter HM, Cooper JS. Acute pulmonary edema after hyper- baric oxygen treatment: a case report written with ChatGPT assistance. Cureus. 2023;15(2):e34752. https://doi.org/10.7759/ cureus.34752. · DOI · Paper page
  35. Manohar N, Prasad SS. Use of ChatGPT in academic publishing: a rare case of seronegative systemic lupus erythematosus in a patient with HIV infection. Cureus. 2023;15(2):e34616. https:// doi.org/10.7759/cureus.34616. · DOI · Paper page
  36. Browne RF, Logan PM, Lee MJ, Torreggiani WC. The accuracy of references in manuscripts submitted for publication. Can Assoc Radiol J. 2004;55(3):170-173.
  37. O'Connor AE, Lukin W, Eriksson L, O'Connor C. Improve- ment in the accuracy of references in the journal Emergency Medicine Australasia. Emerg Med Australas. 2013;25(1):64-67. https://doi.org/10.1111/1742-6723.12030. · DOI · Paper page
  38. Montenegro TS, Hines K, Partyka PP, Harrop J. Reference accu- racy in spine surgery. J Neurosurg Spine. 2020;34(1):1-5. https:// doi.org/10.3171/2020.6.SPINE20640. · DOI · Paper page
  39. Al-Benna S, Rajgarhia P, Ahmed S, Sheikh Z. Accuracy of refer- ences in burns journals. Burns. 2009;35(5):677-680. https://doi. org/10.1016/j.burns.2008.11.014. · DOI · Paper page
  40. Glick M. You are what you cite: the role of references in scien- tific publishing. 12, 14. J Am Dent Assoc. 2007;138(1). https://doi. org/10.14219/jada.archive.2007.0002. · DOI
  41. Gasparyan AY, Yessirkepov M, Voronov AA, Gerasimov AN, Kostyukova EI, Kitas GD. Preserving the integrity of citations and references by all stakeholders of science communication. J Korean Med Sci. 2015;30(11):1545-1552. https://doi.org/10. 3346/jkms.2015.30.11.1545. · DOI
  42. Korngiebel DM, Mooney SD. Considering the possibilities and pitfalls of generative pre-trained transformer 3 (GPT-3) in healthcare delivery. NPJ Digit Med. 2021;4(1):93. https://doi. org/10.1038/s41746-021-00464-x. · DOI

Cited by

Sorted by citations

Authors
Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E.
Affiliations
  • Centre Hospitalier Universitaire Sainte-Justine (CA)
  • Cegep de Saint Hyacinthe (CA)
  • Montreal Children's Hospital (CA)
Corresponding author
Gravel, Jocelyn
ORCID
Venue
Mayo Clinic Proceedings Digital Health
Publisher
openRxiv
Published
2023-03-24
Volume
1
Issue
3
Pages
226-234
ISSN
2949-7612 (ISSN-L)
DOI
10.1101/2023.03.16.23286914
PMC
PMC11975740
OpenAlex
W4380360840
Cited by
220
References
0
Citations by year
2026: 61 · 2025: 56 · 2024: 74 · 2023: 28 · 2013: 1
FWCI
8.75
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Mayo Clinic Proceedings Digital Health (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
ChatGPT, large language models, medical question answering, citation fraud, factual errors, answer quality, AI-generated content, scientific writing, medical manuscripts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review