The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 10 of the 13 sources its claims cite were checked for support.
Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study
Open accessPublished version of a preprint
Abstract
BACKGROUND: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. OBJECTIVE: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. METHODS: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. RESULTS: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. CONCLUSIONS: ChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.
Overview
CoChat Summary
TL;DR
In a cross-disciplinary test, ChatGPT (GPT-3.5) generated 102 citations across 10 natural sciences and humanities topics; citation existence, accuracy, and relevance were similar across domains, but DOI existence and accuracy were significantly worse in the humanities, with more DOI hallucination. The authors conclude that LLM citation reliability varies by discipline and requires validation.
Key findings
- Across 10 topics (5 natural sciences, 5 humanities), ChatGPT generated 102 citations: 55 in natural sciences and 47 in humanities. Citation existence was 40/55 (72.7%) in natural sciences vs 36/47 (76.6%) in humanities (P=.42); citation accuracy was 37/55 (67.3%) vs 29/47 (61.7%) (P=.35); relevance was 39/55 (70.9%) vs 35/47 (74.5%) (P=.43).
- DOI existence differed significantly: 39/55 (70.9%) in natural sciences vs 18/47 (38.3%) in humanities (P=.001). DOI accuracy was 18/55 (32.7%) vs 4/47 (8.5%) (P=.003).
- DOI hallucination was more prevalent in the humanities: Table 2 reports 42/47 (89.4%) in humanities vs 34/55 (61.8%) in natural sciences (P=.001). The abstract states “42/55, 89.4%” for humanities, which conflicts with Table 2’s denominator.
- Levenshtein distance was reported as significantly different (P=.009), but the direction is inconsistent: Table 2 and the Results text report a higher mean in natural sciences (64.13, SD 42.26) than humanities (42.15, SD 40.23), whereas the abstract says it was higher in the humanities.
Methods
- Two researchers independently prompted ChatGPT (GPT-3.5) to write introduction sections for 10 English-language topics (5 clinical/biomedical natural sciences; 5 philosophy/psychology humanities), adhering to APA style and including DOIs, between July 10 and August 15, 2023.
- References were validated via Google Scholar, PubMed, and Google Search; DOI existence and validity were checked using the DOI Foundation website. Hallucination was defined as a nonexistent DOI/citation or a DOI linked to a different source; disagreements were resolved by discussion and consensus.
- Analysis used SPSS 26 and Python; Levenshtein distance was calculated with the thefuzz package. Continuous variables were reported as mean (SD), categorical variables as n (%), with independent-sample t tests and Fisher exact tests; P<.05 was significant.
Limitations
- Stated: only 10 topics were included, limiting coverage of disciplinary complexity; only ChatGPT 3.5 was tested, so newer models (GPT-4, Claude+, Gemini) may differ; the study did not assess user feedback or subjective quality/coherence of generated content.
- Evident from the text: the abstract and Table 2/Results disagree on the direction of the Levenshtein distance finding and on the denominator for humanities DOI hallucination, which complicates interpretation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 3 Not valid · 1 Not checked · 25 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“where the natural sciences literature has often been more proactive in adopting the DOI system of referencing and linking to scholarly works than the humanities.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Discussion > Principal Findings
What the sources say
- Could not verifycites [21] Eve, MP, 2014, Open Access and the Humanities sourceExperimental — accuracy not yet measured
The ‘Open access: scientific origins’ section (page unavailable) discusses differences between humanities and science publishing and says citation serves broadly similar purposes in both fields. It does not address comparative adoption of the DOI system. Searches for DOI-based referencing, digital object identifiers, and disciplinary adoption found no passage establishing the claim. (LLM, conf=0.73)
- Could not verifycites [22] sourceExperimental — accuracy not yet measured
The Introduction discusses slower adoption of open access among humanities researchers, and section 2.3 discusses disciplinary differences in social-media use. Neither addresses adoption of the DOI system for referencing and linking scholarly works; searches for the DOI system and digital object identifiers found no relevant passage. (LLM, conf=0.75)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Although prior works are quite informative [9-12], there is a lack of an interdisciplinary perspective on citations and references generated by LLMs, which is vital for understanding how LLMs perform across different disciplines.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Introduction > Background
What the sources say
- Not checked — no full text of the sourcecites [9] source
- Could not verifycites [10] sourceExperimental — accuracy not yet measured
The paper’s Abstract discusses ChatGPT’s use in literature reviews and debates over citing ChatGPT as an author, but does not address whether research on LLM-generated citations and references lacks an interdisciplinary perspective. Searches for that gap and for comparisons across disciplines found no supporting passage. In the citing sentence, the citations also attach to the description of prior works, while the gap appears to be the citing authors’ own assessment. (LLM, conf=0.76)
- Supportedcites [11] source
The citation is attached to the description of prior works as informative, not to the citing authors’ assessment that an interdisciplinary perspective is lacking. In the paper’s main text (indexed as Abstract; page not specified), Zheng and Zhan examine five ChatGPT responses and warn that writers must check ChatGPT-generated references, making it a relevant prior work on the topic. (LLM, conf=0.86)
- Supportedcites [12] source
The citation is attached to “prior works are quite informative,” not to the citing authors’ claim about a lack of interdisciplinary perspective. In the source’s “PMID: PubMed ID” section, Alkaissi and McFarlane examine ChatGPT-generated references and report PMID numbers belonging to different papers, making it a relevant prior work on the topic. The interdisciplinary gap is the citing authors’ own assessment. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no significant difference between the two groups (P=.42), indicating that the validity of the citations was relatively consistent between the two domains.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > Citation Existence and Accuracy
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, the level of DOI accuracy was significantly higher for the natural sciences than for the humanities (P=.003).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean Levenshtein distance, which measures the deviation between the generated DOI and the actual DOI, was significantly higher in the natural sciences group than in the humanities (P=.009; Table 2).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
8 more claims
Create a free account to see all 31 claims and every source verdict.
- It may state two separate claims as one. 11
- Probably not a substantial claim. 7
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their analysis showed hallucination rates exceeding 80% in prompts referencing lower-income countries.”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, prior studies have examined at most 4 models simultaneously, and none have compared current-generation systems from all major commercial providers within a single experimental framework [8 -10].”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These findings collectively demonstrate that both the disciplinary domain and the specificity of the prompt can influence the likelihood and nature of citation hallucinations, highlighting the need for systematic evaluations of LLM citation reliability across varied research contexts.”
From Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The fundamental gap in the current literature is the lack of a unified perspective that simultaneously measures core factual accuracy alongside bibliographic hallucination across varying prompt complexities.”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“fundamentally driven by data availability bias rather than cognitive weakness”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [12]
Experimental — accuracy not yet measured
14 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Abstract > Results
“The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Results > DOI Existence, Accuracy, and Hallucination
“The mean Levenshtein distance, which measures the deviation between the generated DOI and the actual DOI, was significantly higher in the natural sciences group than in the humanities (P=.009; Table 2).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“DOI hallucination was more prevalent in the humanities (42/55, 89.4%).”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Abstract > Results
Reported 89.4 · recomputed 76.363636
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > Introduction > LLM Concepts
“In 2015, Bahdanau et al [14] proposed accounting for attention to improve RNN performance with long text.”
§ Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study > References
“14. Bahdanau D Cho K Bengio Y Neural machine translation by jointly learning to align and translate arXiv Preprint posted on September 1, 2014.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“Although prior works are quite informative [9-12], there is a lack of an interdisciplinary perspective on citations and references generated by LLMs, which is vital for understanding how LLMs perform across different disciplines.”
A Normalized Levenshtein Distance Metric
Yujian, L., & Bo, L. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1078 · Cited by 986
Encyclopedia of Library and Information Sciences
Bates, M. J., & Maack, M. N. (2009) · doi:10.1201/9780203757635 · Cited by 729
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 328 · cited 1×
“Studies on the viability of LLMs as writing assistants in scholarly writing [6-8] underscore the significance of this body of research within the broader academic landscape.”
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
Could AI help you to write your next paper?
Hutson, M. (2022) · Nature · doi:10.1038/d41586-022-03479-w · Cited by 236 · cited 1×
“These LLMs, powered by vast corpora of text data and sophisticated machine-learning algorithms, have offered researchers and writers a new tool for assistance in crafting scholarly documents [1-3].”
Open Access and the Humanities: Contexts, Controversies and the Future
Eve, M. P. (2014) · doi:10.1017/cbo9781316161012 · Cited by 223
HS Kumar, A. (2023) · Biology, Engineering, Medicine and Science Reports · doi:10.5530/bems.9.1.5 · Cited by 165
Artificial intelligence in academic writing: a paradigm-shifting technological advance
Golan, R., Reddy, R., Muthigi, A., & Ramasamy, R. (2023) · Nature Reviews Urology · doi:10.1038/s41585-023-00746-x · Cited by 136
Research works printed but not linked: 4
- Vaswani, A, arXiv
- Bahdanau, D, arXiv
- DOI Foundation2024-04-01https://doi.org/
- Zhang, Y, arXiv
Cited by
Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses
Luo, X., Chen, F., Zhu, D., Wang, L., Wang, Z., Liu, H., Lyu, M., Wang, Y., Wang, Q., & Chen, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/56780 · Cited by 109
False authorship: an explorative case study around an AI-generated article published under my name
Spinellis, D. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00165-z · Cited by 37
“Furthermore, machine-generated articles pollute the scientific communication landscape with trite platitudes, potentially incorrect facts [16], biased or discriminatory text ([17], p. 33), and often incorrect citations [18].”
Linardon, J., Jarman, H. K., McClure, Z., Anderson, C., Liu, C., & Messer, M. (2025) · JMIR Mental Health · doi:10.2196/80371 · Cited by 40
“Mugaanyi et al [8] compared citation fabrication rates in GPT-3.”
Sun, S. H., Huynh, K. N., Cortes, G., Hill, R., Tran, J., Yeh, L., Ngo, A. L., Houshyar, R., Yaghmai, V., & Tran, M. D. (2024) · Radiology · doi:10.1148/radiol.232346 · Cited by 36
Estimating the use of ChatGPT in dental research publications
Uribe, S. E., & Maldupa, I. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105275 · Cited by 27
Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework
Chang, Y., Yin, J., Li, J., Liu, C., Cao, L., & Lin, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02132-5 · Cited by 23
Generative Artificial Intelligence, AI for Scientific Writing: A Literature ReviewPreprint
Shopovski, J. (2024) · Preprints.org · doi:10.20944/preprints202406.0011.v1 · Cited by 34
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
Rao, D., Wong, E., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2604.03173 · Cited by 16
Using Artificial Intelligence for Scholarly Writing
Oermann, M. H., Owens, J. K., Carter‐Templeton, H., Peterson, G. M., & Bailey, H. E. (2025) · AJN American Journal of Nursing · doi:10.1097/ajn.0000000000000179 · Cited by 20
Autonomous artificial intelligence, scientific research, and human values
Resnik, D. B., Hosseini, M., & Hauswald, R. (2026) · AI and Ethics · doi:10.1007/s43681-025-00908-0 · Cited by 15
“35 For this to happen, we would need to consider the AI system as being conscious and self-aware and having thoughts, feelings, wants, hopes, fears, dreams, and so on (which it does not yet have the capacity for).”
Rewriting Content with GPT-4 to Support Emerging Readers in Adaptive Mathematics Software
Norberg, K. A., Almoubayyed, H., Ley, L. D., Murphy, A., Weldon, K., & RITTER, S. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00420-2 · Cited by 20
Kayabaşı, M., Köksaldı, S., & Engin, C. D. (2024) · Clinical and Experimental Optometry · doi:10.1080/08164622.2024.2419524 · Cited by 14
Is your curriculum GenAI-proof? A method for GenAI impact assessment and a case study
Jongkind, R., Elings, E., Joukes, E., Broens, T., Leopold, H., Wiesman, F., & Meinema, J. G. (2025) · MedEdPublish · doi:10.12688/mep.20815.1 · Cited by 9
Oladokun, B. D., Enakrire, R. T., Emmanuel, A. K., Ajani, Y. A., & Adetayo, A. J. (2025) · Journal of Web Librarianship · doi:10.1080/19322909.2025.2482093 · Cited by 7
Kim, E., Kipchumba, F., & Min, S. (2025) · Publications · doi:10.3390/publications13040049 · Cited by 7
Large language models provide discordant information compared to ophthalmology guidelines
Taloni, A., Sangregorio, A. C., Alessio, G., Romeo, M. A., Coco, G., Busin, L. M. L., Sollazzo, A., Scorcia, V., & Giannaccare, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-06404-z · Cited by 8
See, Y. K. C., Lim, K. S. A., Yung, A. W., Charlene, C. S. Y., Fan, X., & Li, K. Z. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9060151 · Cited by 9
Large language model AI-guided creative writing co-creation in secondary schools
Spitsberg, T., Kettler, T., & McKamie, J. (2025) · Theory Into Practice · doi:10.1080/00405841.2025.2528551 · Cited by 4
Goto, J., & Ramnarain, U. D. (2026) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-026-00588-8 · Cited by 7
Do large language models know basic facts about journal articles?
Thelwall, M. A. (2026) · Journal of Documentation · doi:10.1108/jd-11-2025-0330 · Cited by 6
Jiao, J. R. (2026) · Journal of Engineering Design · doi:10.1080/09544828.2026.2626883 · Cited by 3
Tsai, C., Lin, Y.-C., Hou, J.-U., Tsai, S., Yeh, P.-C., & Kao, C. (2025) · Digital Health · doi:10.1177/20552076251357468 · Cited by 4
Ülkir, M., & Paslı, B. (2026) · Clinical Anatomy · doi:10.1002/ca.70187 · Cited by 5
Cabezas-Clavijo, Á., & Sidorenko-Bautista, P. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2025-0326 · Cited by 22
Quality and Readability of Large Language Models' Responses to Oral Lichen Planus Patients' FAQs
Polizzi, A., Isola, G., Caponio, V. C. A., Santos‐Silva, A. R., González‐Serrano, J., Albuquerque, R., Brailo, V., Farag, A. M., López‐Jornet, P., Robledo‐Sierra, J., et al. (2026) · Oral Diseases · doi:10.1111/odi.70334 · Cited by 3
You Cannot Search the Literature Using Artificial Intelligence, and This Is Why
Oermann, M. H. (2024) · Nursing Education Perspectives · doi:10.1097/01.nep.0000000000001344 · Cited by 2
Who Owns the Argument? A Practical Test for Author Responsibility in AI ‐Assisted Scholarly Writing
Koo, M., & Lu, M. (2026) · Learned Publishing · doi:10.1002/leap.2066 · Cited by 4
Cross sectional pilot study on clinical review generation using large language models
Luo, Z., Qiao, Y., Xu, X., Li, X., Xiao, M., Aijia, K., Wang, D., Pang, Y., Xie, X., Xie, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01535-z · Cited by 7
Picazo-Sanchez, P., & Ortiz-Martin, L. (2026) · Data · doi:10.3390/data11050122 · Cited by 1
“For example, some authors automatically generated 102 references and concluded that over 75% of the generated DOI exist [30].”
Bai, J., Ji, X., Yu, J., Wang, Y., Guo, Y., Xue, C., Zhang, W., & Zhu, J. (2025) · JMIR AI · doi:10.2196/79153 · Cited by 2
Çamlar, M., Sevgi, U. T., Erol, G., Karakaş, F., Doğruel, Y., & Güngör, A. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06628-y · Cited by 4
Assessing the Reliability of ChatGPT and Gemini in Identifying Relevant Orthodontic Literature
Asiri, S. N. (2025) · European Journal of General Dentistry · doi:10.1055/s-0045-1809617 · Cited by 3
Human-AI Narrative Synthesis to Foster Shared Understanding in Civic Decision-Making
Overney, C., Jiang, H., Haider, U., Moe, C., Mangat, J., Pantano, F., McMillian, E. G., Riggins, P., & Gillani, N. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791570 · Cited by 5
Leveraging generative artificial intelligence errors to teach appropriate citation usage
Shah, R., & Miranda, J. L. (2026) · Journal of Microbiology and Biology Education · doi:10.1128/jmbe.00195-25 · Cited by 2
Knowledge graphs and large language models for prompt-based scientometric inquiry
Correia, A., Saarela, M., & Kärkkäinen, T. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104882 · Cited by 3
Chen, C. K., Liang, S.-H., & Wang, T. (2026) · Scientometrics · doi:10.1007/s11192-026-05727-6 · Cited by 1
Zhou, M., Gui, F., Ai, C., Ling, L., Zhang, X., Lu, Y., Han, D., Zhao, B., Zhong, F., Liu, J., et al. (2026) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-025-07092-1 · Cited by 3
Explainable and Interactive Scientometrics with Large Language Models and Knowledge Graphs
Saarela, M., Correia, A., & Kärkkäinen, T. (2025) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit67332.2025.11268107 · Cited by 4
Hallucinations in large language models within academic contexts: A systematic review.
Lai, K. E., Mustaffa, N., Preece, C. N., & Wong, F. Y. (2026) · Journal of Science and Technology Policy Management · doi:10.1108/jstpm-01-2026-0015 · Cited by 1
Naser, M. (2026) · arXiv · doi:10.48550/arxiv.2603.03299 · Cited by 1
“Mugaanyi et al. [9] pursued a geographic dimension by validating 3,451 citations generated by four models using the CrossRef API.”
Hussein, M. A., Doshi, R., He, L., & Reynolds, T. L. (2026) · medRxiv · doi:10.64898/2026.07.15.26355314
Think like an LLM: testing three AI citation verification prompts
Williamson, J. M. (2026) · Library Hi Tech News · doi:10.1108/lhtn-04-2026-0085
(2025) · doi:10.2196/preprints.80371 · Cited by 39
Pramudya, A., Haryanto, V. C. P., Ramli, N. V., Saputra, C. H., Hartono, S., & Reyes, I. V. P. D. (2026) · International Conference on Information Management and Technology (ICIMTech) · doi:10.1109/icimtech202671078.2026.11698921
Guidance Regarding the Use of Artificial Intelligence in Nursing Journal Author Guidelines
Carter‐Templeton, H., Oermann, M. H., Owens, J. K., Peterson, G. M., Mbadiwe, J., Quazi, M. A., & Bailey, H. E. (2025) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001286 · Cited by 1
Liu, S.-W., Chu, S., Wang, H.-C., & Lee, M.-S. (2026) · JMIR Formative Research · doi:10.2196/98343
Citation Inaccuracies and the Need for Multi-Level Oversight in AI-Assisted Medical Writing
Rajaratnam, V., Omar, U. F., Kee, K., & Kaliya-Perumal, A.-K. (2026) · Standards · doi:10.3390/standards6010010 · Cited by 3
Vassileva, J. (2026) · International Conference on Revolutionary Artificial Intelligence and Future Applications (Rev-AI) · doi:10.1109/rev-ai70456.2026.11622089
Lin, F. J.-M., & Cho, S. (2026) · JMIR Formative Research · doi:10.2196/88008
Ikhtiar, I., Fidiana, F., & Islamiyah, W. R. (2025) · Journal of Neurosciences in Rural Practice · doi:10.25259/jnrp_401_2024
Soldatkina, M. A., Tsymbarovich, P., Fomin, D. S., & Ivanov, A. L. (2026) · Agronomy Journal · doi:10.1002/agj2.70324 · Cited by 2
Alghamdi, L. H., & Alghizzi, T. M. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1908592
Sun, M., & Zhu, H. (2026) · IEEE Access · doi:10.1109/access.2026.3735019
Santhosh, V. N., Vas, R., Roychowdhury, B., Sakthi, K., & Rahaman, M. (2026) · Research Evaluation · doi:10.1093/reseval/rvag024
Tariciotti, L., Zohdy, Y. M., Riva, M., Levi, R., Pessina, F., & Pradilla, G. (2025) · Springer eBooks · doi:10.1007/978-3-031-81850-9_21
Soyugür, B. (2026) · Spor Bilimleri Dergisi Hacettepe Üniversitesi · doi:10.17644/sbd.1740656
Oladokun, B. D., Ogunjimi, B. E., Olatunbosun, I. E., Adefila, E. K., Abdul, A., Ebhonu, S. I., Omoniyi, Y. M., & Enakrire, R. T. (2025) · Journal of Library Metadata · doi:10.1080/19386389.2025.2598503 · Cited by 3
Tayo-Ladega, O. T., Dada, O. S., Anih, D. C., & Okorocha, U. C. (2026) · Trends in Social Sciences · doi:10.21124/tss.2026.57.68
Hunt, D., & Miceli, M. D. (2026) · American Journal of STEM Education · doi:10.32674/tfptev94 · Cited by 1
10 insights for harnessing artificial intelligence in scientific research: supervisor perspectives
Mitra, N. K., Vishnumukkala, T., & Chitra, E. (2026) · MedEdPublish · doi:10.12688/mep.21799.1
Jung, E., Yoo, J., Kim, S. G., & Kim, Y. S. (2026) · Gastroenterology report · doi:10.1093/gastro/goag098
Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions
Cook, R., Kahan, A. A., Scharfenberger, T., Tasoulas, J., Hawks-Ladds, N., Chouake, R. J., Jariwala, S. P., & Arora, S. (2026) · Applied Clinical Informatics · doi:10.1055/a-2835-4634 · Cited by 1
Moulaison, H. L., & Thach, H. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1334 · Cited by 2
Badilla, C. A. C. (2026) · Biblios Journal of Librarianship and Information Science · doi:10.47456/biblios.2026.1425
Reconciling Efficiency and Scientific Integrity Through Human-Centered LLM Integration
Holzer, T., & Michahelles, F. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-34044-3_7
Is your curriculum GenAI-proof? A method for GenAI impact assessment and a case study
Jongkind, R., Elings, E., Joukes, E., Broens, T., Leopold, H., Wiesman, F., & Meinema, J. G. (2026) · MedEdPublish · doi:10.12688/mep.20815.2
Farrahi, V. (2026) · Journal for the Measurement of Physical Behaviour · doi:10.1123/jmpb.2025-0066
A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants
Anantama, R., Bachtiar, M. I., & Rahman, Z. (2026) · Bulletin of Data Science · doi:10.47065/bulletinds.v5i3.10443
“The most critical findings regarding cross-disciplinary disparities are revealed in Table 3, synthesizing the findings of Mugaanyi et al. [12].”
Influential
Metadata
- Authors
- Mugaanyi, J., Cai, L., Cheng, S., Lu, C., & Huang, J.
- Affiliations
- Ningbo University (CN)
- Shanghai Academy of Social Sciences (CN)
- Corresponding author
- Huang, Jing
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-04-05
- Volume
- 26
- Pages
- e52935
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11031695
- OpenAlex
- W4392710953
- Cited by
- 74
- References
- 17
- Citations by year
- 2026: 45 · 2025: 21 · 2024: 8
- FWCI
- 109.16
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- large language models, ChatGPT, citation accuracy, reference generation, digital object identifiers, hallucinations, academic writing, Levenshtein distance, AI writing tools, cross-disciplinary comparison
- MeSH terms
- Artificial Intelligence, Humans, Language, Research Personnel, Writing, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
