The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index; 29 of the 64 sources its claims cite were checked for support.
Fabrication and errors in the bibliographic citations generated by ChatGPT
Open accessAbstract
Although chatbots such as ChatGPT can facilitate cost-effective text generation and editing, factually incorrect responses (hallucinations) limit their utility. This study evaluates one particular type of hallucination: fabricated bibliographic citations that do not represent actual scholarly works. We used ChatGPT-3.5 and ChatGPT-4 to produce short literature reviews on 42 multidisciplinary topics, compiling data on the 636 bibliographic citations (references) found in the 84 papers. We then searched multiple databases and websites to determine the prevalence of fabricated citations, to identify errors in the citations to non-fabricated papers, and to evaluate adherence to APA citation format. Within this set of documents, 55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real (non-fabricated) GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors. Although GPT-4 is a major improvement over GPT-3.5, problems remain.
Overview
CoChat Summary
## TL;DR
This study tested how often ChatGPT fabricates bibliographic citations and how often its citations to real works contain errors. Across 636 citations in 84 short papers, GPT-4 improved over GPT-3.5 but still produced substantial fabrication and citation-error rates.
## Key findings
- In 84 ChatGPT-generated papers on 42 topics, the 636 citations were split between GPT-3.5 and GPT-4 outputs; 93% of cited works were scholarly rather than popular.
- Of the 222 works cited in the GPT-3.5 papers, 55% were fabricated; for GPT-4, only 18% were fabricated.
- Among real citations, 43% of GPT-3.5 citations had one or more substantive citation errors, compared with 24% for GPT-4.
- More than 40% of citations had minor APA formatting errors, and fewer than 10% of citations to real works included hyperlinks; when hyperlinks were present in real citations, about one-third were inaccurate.
## Methods
- GPT-3.5 and GPT-4 each generated 42 short literature-review papers from prompts on multidisciplinary topics, yielding 84 papers total and 636 cited works.
- The authors searched Google, Google Scholar, Amazon, DOAJ, PubMed, Scopus, WorldCat, publisher/journal sites, and other sources to determine whether each citation was real or fabricated.
- For real works, they checked substantive citation errors, APA-format adherence, and whether hyperlinks were present and led to the cited work.
## Limitations
- The paper states that it cannot know what proportion of users are taking advantage of GPT-4’s improved performance because detailed information on ChatGPT use is not available.
- The papers were short and generated in response to a specific prompt style, so the results may not generalize beyond this setup; this is evident from the text.
- The study notes that any text after the initial bibliography was excluded, which may omit additional citation behavior; this is evident from the methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 0 Not valid · 11 Not checked · 49 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Several authors have noted that ChatGPT tends to cite works that do not actually exist12,16–25”
§ Fabrication and errors in the bibliographic citations generated by ChatGPT > Introduction > Previous research
What the sources say
- Partly supportedcites [12] Alkaissi, H. & McFarlane, S. I. Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15, e35179. https://doi.org/10.7759/cureus.35179 (2023). sourceExperimental — accuracy not yet measured
The Figure 2 caption and the PMID: PubMed ID section report that ChatGPT supplied references whose PMID numbers belonged to different papers, and reused a list with altered years. This supports the concern about unreliable citations, but the copy does not establish that the cited works themselves did not exist or that ChatGPT tends to cite nonexistent works. (LLM, conf=0.82)
- Supportedcites [16] Eysenbach, G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: A conversation with ChatGPT and a call for papers. JMIR Med. Educ. 9, e46885. https://doi.org/10.2196/46885 (2023). sourceExperimental — accuracy not yet measured
The Commentary reports that ChatGPT invented a reference to a paper that does not exist and supplied unrelated DOIs. This supports citing Eysenbach as one author who noted the problem. (LLM, conf=0.92)
- Not matchedcites [17] Fried, E. I. Using GPT-3 to search for scientific “references”. https://eiko-fried.com/using-gpt-3-to-search-scientific-references/ (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [18] Frosolini, A., Gennaro, P., Cascino, F. & Gabriele, G. In reference to “Role of ChatGPT in public health”, to highlight the AI’s incorrect reference generation. Ann. Biomed. Eng. https://doi.org/10.1007/s10439-023-03248-4 (2023). source
- Supportedcites [19] Halaweh, M. ChatGPT in education: Strategies for responsible implementation. Contemp. Educ. Technol. 15, ep421. https://doi.org/10.30935/cedtech/13036 (2023). sourceExperimental — accuracy not yet measured
In the Reflection note/report (page not provided), Halaweh reports that ChatGPT supplied inaccurate journal references. The author checked one cited article and could not find it in Google, Google Scholar, the journal’s website, or its relevant issues, supporting the claim that ChatGPT can cite works that do not exist. (LLM, conf=0.85)
- Supportedcites [20] Kim, S. G. Using ChatGPT for language editing in scientific articles. Maxillofac. Plast. Reconstr. Surg. 45, 13. https://doi.org/10.1186/s40902-023-00381-x (2023). sourceExperimental — accuracy not yet measured
In the article’s main text, Kim reports checking references supplied by ChatGPT and finding that none of them, including the authors, were real. The paper therefore supports its role as one of the cited accounts noting fabricated references. (LLM, conf=0.88)
- Supportedcites [21] King, M. R. A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell. Mol. Bioeng. 16, 1–2 (2023). sourceExperimental — accuracy not yet measured
In the main text, King asks ChatGPT to generate references on chatbots, AI, and plagiarism. The References section then lists the generated entries and includes an author’s note that they are not real, directly illustrating the claim about nonexistent cited works. (LLM, conf=0.88)
- Supportedcites [22] Sanchez-Ramos, L., Lin, L. & Romero, R. Beware of references when using ChatGPT as a source of information to write scientific articles. Am. J. Obstet. Gynecol. https://doi.org/10.1016/j.ajog.2023.04.004 (2023). sourceExperimental — accuracy not yet measured
In the letter’s main text (p. 1), the authors report that ChatGPT frequently produces fictitious references, alongside references with incorrect bibliographic details. This supports the claim that it tends to cite works that do not exist. (LLM, conf=0.93)
- Supportedcites [23] Tafferner, Z., Balázs, I., Krammer, O. & Géczy, O. Can ChatGPT help in electronics research and development? A case study with applied sensors. Sensors (Basel) 23, 4879. https://doi.org/10.3390/s23104879 (2023). sourceExperimental — accuracy not yet measured
The Results (section 3.4) describe ChatGPT recommending citations to articles that did not exist. The Discussion (section 4) says the citations it produced for the tested technical themes were fabricated, supporting the claim that it can cite nonexistent works. (LLM, conf=0.93)
- Not matchedcites [24] Weiser, B. Here’s what happens when your lawyer uses ChatGPT. N.Y. Times Web. https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT often provides incorrect responses when asked “Is this citation correct” or “Do you fabricate citations?”12,13,18,24,29.”
§ Fabrication and errors in the bibliographic citations generated by ChatGPT > Introduction > Previous research
What the sources say
- Partly supportedcites [12] Alkaissi, H. & McFarlane, S. I. Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15, e35179. https://doi.org/10.7759/cureus.35179 (2023). sourceExperimental — accuracy not yet measured
The PMID: PubMed ID section reports that ChatGPT supplied references with PMID numbers belonging to different papers, supporting the concern about inaccurate citations. The paper does not report testing its answers to the specific questions “Is this citation correct?” or “Do you fabricate citations?”, or establish how often those answers are incorrect. (LLM, conf=0.84)
- Partly supportedcites [13] Gravel, J., D’Amours-Gravel, M. & Osmanlliu, E. Learning to fake it: Limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin. Proc. Digit. Health https://doi.org/10.1101/2023.03.16.23286914(24Mar (2023). sourceExperimental — accuracy not yet measured
The Results section reports that, when challenged about reference accuracy, ChatGPT sometimes gave a misleading answer: it said references were available in PubMed but linked to unrelated publications. The paper also found that 41 of 59 evaluated references were fabricated. It does not establish that ChatGPT often answers the specific follow-up questions about citation correctness or its own fabrication incorrectly; it describes its answers to challenges as variable. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [18] Frosolini, A., Gennaro, P., Cascino, F. & Gabriele, G. In reference to “Role of ChatGPT in public health”, to highlight the AI’s incorrect reference generation. Ann. Biomed. Eng. https://doi.org/10.1007/s10439-023-03248-4 (2023). source
- Not matchedcites [24] Weiser, B. Here’s what happens when your lawyer uses ChatGPT. N.Y. Times Web. https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [29] Hueber, A. J. & Kleyer, A. Quality of citation data using the natural language processing tool ChatGPT in rheumatology: Creation of false references. RMD Open 9, e003248. https://doi.org/10.1136/rmdopen-2023-003248 (2023). sourceExperimental — accuracy not yet measured
In the paper’s main text (page unavailable), ChatGPT initially denied that fictitious references were fake when confronted about their authenticity, then acknowledged they were fictitious when asked again. This supports an incorrect response to a citation-verification question, but the paper does not establish that such answers occur often or report the quoted questions as its prompts. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“6 studies have systematically investigated the extent of the problem13,15,26–29.”
§ Fabrication and errors in the bibliographic citations generated by ChatGPT > Introduction > Previous research
What the sources say
- Supportedcites [13] Gravel, J., D’Amours-Gravel, M. & Osmanlliu, E. Learning to fake it: Limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin. Proc. Digit. Health https://doi.org/10.1101/2023.03.16.23286914(24Mar (2023). source
The Results report that the study checked 59 ChatGPT-provided references and found 41 (69%) fabricated. This supports citing the paper as one study that systematically investigated fabricated citations; the citation need not independently establish the total of six studies. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [15] Wagner, M. W. & Ertl-Wagner, B. B. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can. Assoc. Radiol. J. https://doi.org/10.1177/08465371231171125 (2023). source
- Supportedcites [26] Athaluri, S. A. et al. Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus 15, e37432. https://doi.org/10.7759/cureus.37432 (2023). source
The Materials and methods section describes a systematic check of 178 ChatGPT-generated references by five researchers, and the Results report how many could not be verified. This paper therefore supports its role as one of the six cited studies investigating the extent of fabricated citations; it does not itself establish the count of six. (LLM, conf=0.91)
- Supportedcites [27] Bhattacharyya, M., Miller, V. M., Bhattacharyya, D. & Miller, L. E. High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus 15, e39238. https://doi.org/10.7759/cureus.39238 (2023). source
The Materials and methods section describes a systematic check of references in 30 ChatGPT-generated biomedical papers, and the Results report the proportion fabricated. This paper therefore qualifies as one of the studies cited for the sentence; the total of six is the citing authors’ count across the grouped citations, not a finding attributed to this paper alone. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [28] Day, T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. Prof. Geogr. https://doi.org/10.1080/00330124.2023.2190373 (2023). source
- Supportedcites [29] Hueber, A. J. & Kleyer, A. Quality of citation data using the natural language processing tool ChatGPT in rheumatology: Creation of false references. RMD Open 9, e003248. https://doi.org/10.1136/rmdopen-2023-003248 (2023). source
The main text (page unavailable) describes testing ChatGPT citations across three rheumatology topics and reports how many references were fabricated in each. This supports citing the paper as one study investigating the extent of fabricated citations; it does not need to establish the citing sentence’s total of six studies on its own. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Errors in the numerical components of citations are especially common26,27.”
§ Fabrication and errors in the bibliographic citations generated by ChatGPT > Introduction > Previous research
What the sources say
- Partly supportedcites [26] Athaluri, S. A. et al. Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus 15, e37432. https://doi.org/10.7759/cureus.37432 (2023). sourceExperimental — accuracy not yet measured
The Results and Discussion (page unavailable) report that 69 of 178 ChatGPT-generated references had invalid DOIs, including 41 references to real articles whose supplied DOIs were erroneous. This supports frequent errors in citation identifiers, but the paper does not examine numerical citation components generally or show that their errors are especially common relative to other errors. (LLM, conf=0.82)
- Partly supportedcites [27] Bhattacharyya, M., Miller, V. M., Bhattacharyya, D. & Miller, L. E. High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus 15, e39238. https://doi.org/10.7759/cureus.39238 (2023). sourceExperimental — accuracy not yet measured
The Discussion reports that over 90% of references in this study had an incorrect PMID number, supporting the claim that numerical citation errors were common in its ChatGPT-generated medical content. It only raises the possibility that numerical data are more error-prone than textual data; it does not establish that comparison or a general pattern across citations. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“ChatGPT's fabricated citations tend to look legitimate at first glance13,28”
§ Fabrication and errors in the bibliographic citations generated by ChatGPT > Introduction > Previous research
What the sources say
- Supportedcites [13] Gravel, J., D’Amours-Gravel, M. & Osmanlliu, E. Learning to fake it: Limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin. Proc. Digit. Health https://doi.org/10.1101/2023.03.16.23286914(24Mar (2023). source
The Results say the references looked credible, although 41 of 59 were fabricated. The Discussion likewise says most fabricated references seemed deceptively real, supporting the claim that they look legitimate at first glance. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites [28] Day, T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. Prof. Geogr. https://doi.org/10.1080/00330124.2023.2190373 (2023). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
38 more claims · 2 with their cited sources checked
Create a free account to see all 71 claims and to verify the 3 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 13
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs routinely fabricate references that do not exist via several means (e.g., attaching real author names to fictitious titles, assigning plausible paper titles that were never published, etc.) [3].”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“fabricated citations often contained real author names combined with fictitious titles”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“open-ended prompts trigger up to 55% citation fabrication [11]”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have shown that even state-of-the-art models hallucinate references at rates of 18-90% in long-form scientific text [8, 9]”
From Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings collectively demonstrate that both the disciplinary domain and the specificity of the prompt can influence the likelihood and nature of citation hallucinations, highlighting the need for systematic evaluations of LLM citation reliability across varied research contexts.”
From Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study (2025) · cites this paper as [7]
Experimental — accuracy not yet measured
33 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 3×
“These errors are potentially dangerous, and they are exacerbated by the fact that ChatGPT often stands by its incorrect statements when asked to verify them 12,13,18,24,29 .”
BackgroundLund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607
Tools such as ChatGPT threaten transparent science; here are our ground rules for their use
(2023) · Nature · doi:10.1038/d41586-023-00191-1 · Cited by 605
A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education
King, M. R., & chatGPT (2023) · Cellular and Molecular Bioengineering · doi:10.1007/s12195-022-00754-8 · Cited by 602
AI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543
Athaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023) · Cureus · doi:10.7759/cureus.37432 · Cited by 403
High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content
Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023) · Cureus · doi:10.7759/cureus.39238 · Cited by 294
Research works printed but not linked: 3
- DePeau-Wilson, M. Peer-reviewed journal publishes paper written almost entirely by ChatGPT. MedPage Today. https://www.medpagetoday.com/special-reports/exclusives/102960 (2023).
- Atlas, S. Chatbot Prompting: A Guide for Students, Educators, and an AI-Augmented Workforce (S. Atlas, 2023). https://www.researchgate.net/publication/367464129_Chatbot_Prompting_A_guide_for_students_educators_and_an_AI-augmented_workforce.
- Jacsó, P. As we may search: Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Curr. Sci. 89, 1537–1547 (2005).
Cited by
Sun, Y., Sheng, D., Zhou, Z., & Wu, Y. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03811-x · Cited by 327
Liu, M., Zhang, L. J., & Biebricher, C. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104977 · Cited by 282
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228
“Software developers at OpenAI, Google, and other chatbot companies have been trying to fix this problem, but it is not easy to solve, due to differences between human and LLM processing of language [24, 230].”
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213
“However, LLMs are infamous for making up references (Walters and Wilder, 2023), providing inaccurate summaries (Tang et al., 2024a; Kim et al., 2024), and making false claims that sound authoritative and convincing (Mittelstadt et al., 2023; Ji et al., 2023; Zhang et al., 2023; Zheng and Zhan,…”
Metadata
- Authors
- Walters, W. H., & Wilder, E. I.
- Affiliations
- Manhattan University (US)
- The Graduate Center, CUNY (US)
- Corresponding author
- Walters, William H.
- Venue
- Scientific Reports
- Published
- 2023-09-07
- Volume
- 13
- Issue
- 1
- Pages
- 14045-14045
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC10484980
- OpenAlex
- W4386510404
- Semantic Scholar
- 51dd1ebcebafce801b8856c925dd205f738d8f74
- Cited by
- 499
- References
- 31
- Citations by year
- 2026: 249 · 2025: 117 · 2024: 118 · 2023: 12
- FWCI
- 123.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CUNY Academic Works (City University of New York) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- ChatGPT, large language models, hallucinations, bibliographic references, citation errors, GPT-3.5, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fabrication and errors in the bibliographic citations generated by ChatGPT. https://app.cochat.ai/papers/10.1038/s41598-023-41032-5
- The paper
- Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports. https://doi.org/10.1038/s41598-023-41032-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.