The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles
Abstract
Objective To evaluate the reliability of three artificial intelligence (AI) chatbots (ChatGPT, Google Bard, and Chatsonic) in generating accurate references from existing obstetric literature. Study Design Between mid-March and late April 2023, ChatGPT, Google Bard, and Chatsonic were prompted to provide references for specific obstetrical randomized controlled trials (RCTs) published in 2020. RCTs were considered for inclusion if they were mentioned in a previous article that primarily evaluated RCTs published by the top medical and obstetrics and gynecology journals with the highest impact factors in 2020 as well as RCTs published in a new journal focused on publishing obstetric RCTs. The selection of the three AI models was based on their popularity, performance in natural language processing, and public availability. Data collection involved prompting the AI chatbots to provide references according to a standardized protocol. The primary evaluation metric was the accuracy of each AI model in correctly citing references, including authors, publication title, journal name, and digital object identifier (DOI). Statistical analysis was performed using a permutation test to compare the performance of the AI models. Results Among the 44 RCTs analyzed, Google Bard demonstrated the highest accuracy, correctly citing 13.6% of the requested RCTs, whereas ChatGPT and Chatsonic exhibited lower accuracy rates of 2.4 and 0%, respectively. Google Bard often substantially outperformed Chatsonic and ChatGPT in correctly citing the studied reference components. The majority of references from all AI models studied were noted to provide DOIs for unrelated studies or DOIs that do not exist. Conclusion To ensure the reliability of scientific information being disseminated, authors must exercise caution when utilizing AI for scientific writing and literature search. However, despite their limitations, collaborative partnerships between AI systems and researchers have the potential to drive synergistic advancements, leading to improved patient care and outcomes. Key Points
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several studies have shown that while the model can generate plausible-sounding citations, many of these are either partially incorrect or entirely fabricated, often combining real author names with nonexistent titles or invalid DOIs. 25-29”
From Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics (2025) · cites this paper as [27]
In the citing paper
“These hallucinated references have been documented across various disciplines, including medicine, education, and health sciences, raising widespread concerns about the reliability of AI-generated bibliographic content.”
From Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics (2025) · cites this paper as [27]
Citations
Cites
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
ChatGPT for healthcare services: An emerging stage for an innovative perspective
Javaid, M., Haleem, A., & Singh, R. P. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100105 · Cited by 429
Respiratory Syncytial Virus Vaccination during Pregnancy and Effects in Infants
Madhi, S. A., Polack, F. P., Piedra, P. A., Munoz, F. M., Trenholme, A. A., Simões, E. A. F., Swamy, G. K., Agrawal, S., Ahmed, K., August, A., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa1908380 · Cited by 418
Hoffman, M. K., Goudar, S. S., Kodkany, B. S., Metgud, M., Somannavar, M., Okitawutshu, J., Lokangaka, A., Tshefu, A., Bose, C. L., Mwapule, A., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32973-3 · Cited by 360
ChatGPT is shaping the future of medical writing but still requires human judgment
Kitamura, F. C. (2023) · Radiology · doi:10.1148/radiol.230171 · Cited by 317
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234
Using ChatGPT for language editing in scientific articles
Kim, S.-G. (2023) · Maxillofacial Plastic and Reconstructive Surgery · doi:10.1186/s40902-023-00381-x · Cited by 153
A comparison of ChatGPT-generated articles with human-written articles
Ariyaratne, S., Iyengar, K. P., Nischal, N., Chitti Babu, N., & Botchu, R. (2023) · Skeletal Radiology · doi:10.1007/s00256-023-04340-5 · Cited by 148
Pravastatin for early‐onset pre‐eclampsia: a randomised, blinded, placebo‐controlled trial
Ahmed, A., Williams, D. J., Cheed, V., Middleton, L., Ahmad, S., Wang, K., Vince, A., Hewett, P. W., Spencer, K., Khan, K. S., et al. (2019) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16013 · Cited by 147
Antenatal Dexamethasone for Early Preterm Birth in Low-Resource Countries
Collaborators, T. W. A. T. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2022398 · Cited by 147
Cited by
Ng, J. Y. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101222 · Cited by 21
Zhao, M., Zhou, M., Han, Y., Song, X., Zhou, Y., & He, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-19370-3 · Cited by 18
Karaağaç, M., & Çarkıt, S. (2025) · Oncology · doi:10.1159/000546726 · Cited by 7
Safran, E., & Çalı, A. (2025) · Anatolian Current Medical Journal · doi:10.38053/acmj.1746227 · Cited by 4
Ng, J. Y. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1745928 · Cited by 7
“In health and medical research, this can manifest as misrepresented study findings, fictitious references, or misleading summaries of scientific literature (Walters and Wilder, 2023; Chen and Chen, 2023; Graf et al., 2024; Loh, 2023).”
Metadata
- Authors
- Graf, E. M., McKinney, J. A., Dye, A. B., Lin, L., & Sanchez-Ramos, L.
- Affiliations
- University of Florida (US)
- University of Arizona (US)
- Department of Obstetrics and Gynecology, University of Florida College of Medicine, Jacksonville, Florida
- Corresponding author
- McKinney, Jordan A.
- Venue
- American Journal of Perinatology
- Publisher
- Georg Thieme Verlag KG
- Published
- 2024-04-23
- Volume
- 41
- Issue
- 15
- Pages
- 2072-2081
- ISSN
- 0735-1631 (ISSN-L)
- 1098-8785
- OpenAlex
- W4395051595
- Semantic Scholar
- c0d8e76c4998bd86974e45254dbdeba1d1b3c74f
- Cited by
- 12
- References
- 61
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 2
- FWCI
- 1.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Perinatology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- AI chatbots, reference generation, citation accuracy, ChatGPT, Google Bard, digital object identifiers, responsible AI use, scientific writing
- MeSH terms
- Artificial Intelligence, Humans, Natural Language Processing, Obstetrics, Periodicals as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles. https://app.cochat.ai/papers/10.1055/s-0044-1786033
- The paper
- Graf, E. M., McKinney, J. A., Dye, A. B., Lin, L., & Sanchez-Ramos, L. (2024). Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles. American Journal of Perinatology. https://doi.org/10.1055/s-0044-1786033
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.