Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 10 of the 13 sources its claims cite were checked for support.
Abstract
BACKGROUND: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. OBJECTIVE: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. METHODS: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. RESULTS: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. CONCLUSIONS: ChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.
Overview
CoChat Summary
TL;DR
In a cross-disciplinary test, ChatGPT (GPT-3.5) generated 102 citations across 10 natural sciences and humanities topics; citation existence, accuracy, and relevance were similar across domains, but DOI existence and accuracy were significantly worse in the humanities, with more DOI hallucination. The authors conclude that LLM citation reliability varies by discipline and requires validation.
Key findings
- Across 10 topics (5 natural sciences, 5 humanities), ChatGPT generated 102 citations: 55 in natural sciences and 47 in humanities. Citation existence was 40/55 (72.7%) in natural sciences vs 36/47 (76.6%) in humanities (P=.42); citation accuracy was 37/55 (67.3%) vs 29/47 (61.7%) (P=.35); relevance was 39/55 (70.9%) vs 35/47 (74.5%) (P=.43).
- DOI existence differed significantly: 39/55 (70.9%) in natural sciences vs 18/47 (38.3%) in humanities (P=.001). DOI accuracy was 18/55 (32.7%) vs 4/47 (8.5%) (P=.003).
- DOI hallucination was more prevalent in the humanities: Table 2 reports 42/47 (89.4%) in humanities vs 34/55 (61.8%) in natural sciences (P=.001). The abstract states “42/55, 89.4%” for humanities, which conflicts with Table 2’s denominator.
- Levenshtein distance was reported as significantly different (P=.009), but the direction is inconsistent: Table 2 and the Results text report a higher mean in natural sciences (64.13, SD 42.26) than humanities (42.15, SD 40.23), whereas the abstract says it was higher in the humanities.
Methods
- Two researchers independently prompted ChatGPT (GPT-3.5) to write introduction sections for 10 English-language topics (5 clinical/biomedical natural sciences; 5 philosophy/psychology humanities), adhering to APA style and including DOIs, between July 10 and August 15, 2023.
- References were validated via Google Scholar, PubMed, and Google Search; DOI existence and validity were checked using the DOI Foundation website. Hallucination was defined as a nonexistent DOI/citation or a DOI linked to a different source; disagreements were resolved by discussion and consensus.
- Analysis used SPSS 26 and Python; Levenshtein distance was calculated with the thefuzz package. Continuous variables were reported as mean (SD), categorical variables as n (%), with independent-sample t tests and Fisher exact tests; P<.05 was significant.
Limitations
- Stated: only 10 topics were included, limiting coverage of disciplinary complexity; only ChatGPT 3.5 was tested, so newer models (GPT-4, Claude+, Gemini) may differ; the study did not assess user feedback or subjective quality/coherence of generated content.
- Evident from the text: the abstract and Table 2/Results disagree on the direction of the Levenshtein distance finding and on the denominator for humanities DOI hallucination, which complicates interpretation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 3 Not valid · 1 Not checked · 25 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“where the natural sciences literature has often been more proactive in adopting the DOI system of referencing and linking to scholarly works than the humanities.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Discussion > Principal Findings
What the sources say
- Could not verifycites [21] Eve, MP, 2014, Open Access and the Humanities sourceExperimental — accuracy not yet measured
The ‘Open access: scientific origins’ section (page unavailable) discusses differences between humanities and science publishing and says citation serves broadly similar purposes in both fields. It does not address comparative adoption of the DOI system. Searches for DOI-based referencing, digital object identifiers, and disciplinary adoption found no passage establishing the claim. (LLM, conf=0.73)
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
The Introduction discusses slower adoption of open access among humanities researchers, and section 2.3 discusses disciplinary differences in social-media use. Neither addresses adoption of the DOI system for referencing and linking scholarly works; searches for the DOI system and digital object identifiers found no relevant passage. (LLM, conf=0.75)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Although prior works are quite informative [9-12], there is a lack of an interdisciplinary perspective on citations and references generated by LLMs, which is vital for understanding how LLMs perform across different disciplines.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Introduction > Background
What the sources say
- Not checked — no full text of the sourcecites [9] source
- Could not verifycites [10] sourceExperimental — accuracy not yet measured
The paper’s Abstract discusses ChatGPT’s use in literature reviews and debates over citing ChatGPT as an author, but does not address whether research on LLM-generated citations and references lacks an interdisciplinary perspective. Searches for that gap and for comparisons across disciplines found no supporting passage. In the citing sentence, the citations also attach to the description of prior works, while the gap appears to be the citing authors’ own assessment. (LLM, conf=0.76)
- Supportedcites [11] source
The citation is attached to the description of prior works as informative, not to the citing authors’ assessment that an interdisciplinary perspective is lacking. In the paper’s main text (indexed as Abstract; page not specified), Zheng and Zhan examine five ChatGPT responses and warn that writers must check ChatGPT-generated references, making it a relevant prior work on the topic. (LLM, conf=0.86)
- Supportedcites [12] source
The citation is attached to “prior works are quite informative,” not to the citing authors’ claim about a lack of interdisciplinary perspective. In the source’s “PMID: PubMed ID” section, Alkaissi and McFarlane examine ChatGPT-generated references and report PMID numbers belonging to different papers, making it a relevant prior work on the topic. The interdisciplinary gap is the citing authors’ own assessment. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no significant difference between the two groups (P=.42), indicating that the validity of the citations was relatively consistent between the two domains.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > Citation Existence and Accuracy
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, the level of DOI accuracy was significantly higher for the natural sciences than for the humanities (P=.003).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean Levenshtein distance, which measures the deviation between the generated DOI and the actual DOI, was significantly higher in the natural sciences group than in the humanities (P=.009; Table 2).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
8 more claims
Create a free account to see all 31 claims and every source verdict.
- It may state two separate claims as one. 11
- Probably not a substantial claim. 7
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their analysis showed hallucination rates exceeding 80% in prompts referencing lower-income countries.”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, prior studies have examined at most 4 models simultaneously, and none have compared current-generation systems from all major commercial providers within a single experimental framework [8 -10].”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings collectively demonstrate that both the disciplinary domain and the specificity of the prompt can influence the likelihood and nature of citation hallucinations, highlighting the need for systematic evaluations of LLM citation reliability across varied research contexts.”
From Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The fundamental gap in the current literature is the lack of a unified perspective that simultaneously measures core factual accuracy alongside bibliographic hallucination across varying prompt complexities.”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“fundamentally driven by data availability bias rather than cognitive weakness”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [12]
Experimental — accuracy not yet measured
14 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Abstract > Results
“The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
“The mean Levenshtein distance, which measures the deviation between the generated DOI and the actual DOI, was significantly higher in the natural sciences group than in the humanities (P=.009; Table 2).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“DOI hallucination was more prevalent in the humanities (42/55, 89.4%).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Abstract > Results
Reported 89.4 · recomputed 76.363636
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Introduction > LLM Concepts
“In 2015, Bahdanau et al [14] proposed accounting for attention to improve RNN performance with long text.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > References
“14. Bahdanau D Cho K Bengio Y Neural machine translation by jointly learning to align and translate arXiv Preprint posted on September 1, 2014.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“Although prior works are quite informative [9-12], there is a lack of an interdisciplinary perspective on citations and references generated by LLMs, which is vital for understanding how LLMs perform across different disciplines.”
A Normalized Levenshtein Distance Metric
Yujian, L., & Bo, L. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1078 · Cited by 986 · cited 1×
“The Levenshtein distance [20] between the generated DOI and the actual DOI was calculated using the thefuzz package in Python to quantitatively assess the DOI accuracy.”
Encyclopedia of Library and Information Sciences
Bates, M. J., & Maack, M. N. (2009) · doi:10.1201/9780203757635 · Cited by 729
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 328 · cited 1×
“Studies on the viability of LLMs as writing assistants in scholarly writing [6-8] underscore the significance of this body of research within the broader academic landscape.”
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
Could AI help you to write your next paper?
Hutson, M. (2022) · Nature · doi:10.1038/d41586-022-03479-w · Cited by 236 · cited 1×
“These LLMs, powered by vast corpora of text data and sophisticated machine-learning algorithms, have offered researchers and writers a new tool for assistance in crafting scholarly documents [1-3].”
Open Access and the Humanities: Contexts, Controversies and the Future
Eve, M. P. (2014) · doi:10.1017/cbo9781316161012 · Cited by 223
HS Kumar, A. (2023) · Biology, Engineering, Medicine and Science Reports · doi:10.5530/bems.9.1.5 · Cited by 165
Artificial intelligence in academic writing: a paradigm-shifting technological advance
Golan, R., Reddy, R., Muthigi, A., & Ramasamy, R. (2023) · Nature Reviews Urology · doi:10.1038/s41585-023-00746-x · Cited by 136
Artificial intelligence hallucinations
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04473-y · Cited by 136
ChatGPT in Scientific Writing: A Cautionary Tale
Zheng, H., & Zhan, H. (2023) · The American Journal of Medicine · doi:10.1016/j.amjmed.2023.02.011 · Cited by 104
Marchandot, B., Matsushita, K., Carmona, A., Trimaille, A., & Morel, O. (2023) · European Heart Journal Open · doi:10.1093/ehjopen/oead007 · Cited by 99
Esplugas, M. (2023) · Journal of Hand Surgery (European Volume) · doi:10.1177/17531934231185746 · Cited by 68
Scientists used ChatGPT to generate an entire paper from scratch — but is it any good?
Conroy, G. (2023) · Nature · doi:10.1038/d41586-023-02218-z · Cited by 65
Digital Object Identifier (DOI®) System
Paskin, N. (2009) · CRC Press eBooks · doi:10.1081/e-elis3-120044418 · Cited by 42
Narayan, B., Luca, E. J., Tiffen, B., England, A., Booth, M., & Boateng, H. (2018) · Open Information Science · doi:10.1515/opis-2018-0013 · Cited by 35
Scholarly Communication and Machine-Generated Text: Is it Finally AI vs AI in Plagiarism Detection?
Santra, P. P., & Majhi, D. (2023) · Journal of Information and Knowledge · doi:10.17821/srels/2023/v60i3/171028 · Cited by 34
Research works printed but not linked: 4
- Vaswani, A, arXiv
- Bahdanau, D, arXiv
- DOI Foundation2024-04-01https://doi.org/
- Zhang, Y, arXiv
Cited by
Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses
Luo, X., Chen, F., Zhu, D., Wang, L., Wang, Z., Liu, H., Lyu, M., Wang, Y., Wang, Q., & Chen, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/56780 · Cited by 109
False authorship: an explorative case study around an AI-generated article published under my name
Spinellis, D. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00165-z · Cited by 37
“Furthermore, machine-generated articles pollute the scientific communication landscape with trite platitudes, potentially incorrect facts [16], biased or discriminatory text ([17], p. 33), and often incorrect citations [18].”
Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., & Messer, M. (2025) · JMIR Mental Health · doi:10.2196/80371 · Cited by 40
“Mugaanyi et al [8] compared citation fabrication rates in GPT-3.”
Sun, S. H., Huynh, K. N., Cortes, G., Hill, R., Tran, J., Yeh, L., Ngo, A. L., Houshyar, R., Yaghmai, V., & Tran, M. D. (2024) · Radiology · doi:10.1148/radiol.232346 · Cited by 36
Estimating the use of ChatGPT in dental research publications
Uribe, S. E., & Maldupa, I. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105275 · Cited by 27
Metadata
- Authors
- Mugaanyi, J., Cai, L., Cheng, S., Lu, C., & Huang, J.
- Affiliations
- Ningbo University (CN)
- Shanghai Academy of Social Sciences (CN)
- Corresponding author
- Huang, Jing
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-04-05
- Volume
- 26
- Pages
- e52935
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11031695
- OpenAlex
- W4392710953
- Cited by
- 74
- References
- 17
- Citations by year
- 2026: 45 · 2025: 21 · 2024: 8
- FWCI
- 109.16
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- large language models, ChatGPT, citation accuracy, reference generation, digital object identifiers, hallucinations, academic writing, Levenshtein distance, AI writing tools, cross-disciplinary comparison
- MeSH terms
- Artificial Intelligence, Humans, Language, Research Personnel, Writing, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
