CoChat

Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles

Graf, Emily M., McKinney, Jordan A., Dye, Alexander B., Lin, Lifeng, Sanchez-Ramos, Luis · 2024 · American Journal of Perinatology

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.

Abstract

Objective To evaluate the reliability of three artificial intelligence (AI) chatbots (ChatGPT, Google Bard, and Chatsonic) in generating accurate references from existing obstetric literature. Study Design Between mid-March and late April 2023, ChatGPT, Google Bard, and Chatsonic were prompted to provide references for specific obstetrical randomized controlled trials (RCTs) published in 2020. RCTs were considered for inclusion if they were mentioned in a previous article that primarily evaluated RCTs published by the top medical and obstetrics and gynecology journals with the highest impact factors in 2020 as well as RCTs published in a new journal focused on publishing obstetric RCTs. The selection of the three AI models was based on their popularity, performance in natural language processing, and public availability. Data collection involved prompting the AI chatbots to provide references according to a standardized protocol. The primary evaluation metric was the accuracy of each AI model in correctly citing references, including authors, publication title, journal name, and digital object identifier (DOI). Statistical analysis was performed using a permutation test to compare the performance of the AI models. Results Among the 44 RCTs analyzed, Google Bard demonstrated the highest accuracy, correctly citing 13.6% of the requested RCTs, whereas ChatGPT and Chatsonic exhibited lower accuracy rates of 2.4 and 0%, respectively. Google Bard often substantially outperformed Chatsonic and ChatGPT in correctly citing the studied reference components. The majority of references from all AI models studied were noted to provide DOIs for unrelated studies or DOIs that do not exist. Conclusion To ensure the reliability of scientific information being disseminated, authors must exercise caution when utilizing AI for scientific writing and literature search. However, despite their limitations, collaborative partnerships between AI systems and researchers have the potential to drive synergistic advancements, leading to improved patient care and outcomes. Key Points

Overview

Integrity signals

Registry

Crossref · 10.1055/s-0044-1786033

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Cited by

See all 12

Metadata

Authors
Graf, E. M., McKinney, J. A., Dye, A. B., Lin, L., & Sanchez-Ramos, L.
Affiliations
  • University of Florida (US)
  • University of Arizona (US)
  • Department of Obstetrics and Gynecology, University of Florida College of Medicine, Jacksonville, Florida
Corresponding author
McKinney, Jordan A.
Venue
American Journal of Perinatology
Published
2024-04-23
Volume
41
Issue
15
Pages
2072-2081
ISSN
  • 0735-1631 (ISSN-L)
  • 1098-8785
OpenAlex
W4395051595
Cited by
12
References
61
Citations by year
2026: 5 · 2025: 5 · 2024: 2
FWCI
1.12
Type
journal article
Language
en
Open access
closed
Locations
  • American Journal of Perinatology (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
AI chatbots, reference generation, citation accuracy, ChatGPT, Google Bard, digital object identifiers, responsible AI use, scientific writing
MeSH terms
Artificial Intelligence, Humans, Natural Language Processing, Obstetrics, Periodicals as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.