Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles
The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Objective To evaluate the reliability of three artificial intelligence (AI) chatbots (ChatGPT, Google Bard, and Chatsonic) in generating accurate references from existing obstetric literature. Study Design Between mid-March and late April 2023, ChatGPT, Google Bard, and Chatsonic were prompted to provide references for specific obstetrical randomized controlled trials (RCTs) published in 2020. RCTs were considered for inclusion if they were mentioned in a previous article that primarily evaluated RCTs published by the top medical and obstetrics and gynecology journals with the highest impact factors in 2020 as well as RCTs published in a new journal focused on publishing obstetric RCTs. The selection of the three AI models was based on their popularity, performance in natural language processing, and public availability. Data collection involved prompting the AI chatbots to provide references according to a standardized protocol. The primary evaluation metric was the accuracy of each AI model in correctly citing references, including authors, publication title, journal name, and digital object identifier (DOI). Statistical analysis was performed using a permutation test to compare the performance of the AI models. Results Among the 44 RCTs analyzed, Google Bard demonstrated the highest accuracy, correctly citing 13.6% of the requested RCTs, whereas ChatGPT and Chatsonic exhibited lower accuracy rates of 2.4 and 0%, respectively. Google Bard often substantially outperformed Chatsonic and ChatGPT in correctly citing the studied reference components. The majority of references from all AI models studied were noted to provide DOIs for unrelated studies or DOIs that do not exist. Conclusion To ensure the reliability of scientific information being disseminated, authors must exercise caution when utilizing AI for scientific writing and literature search. However, despite their limitations, collaborative partnerships between AI systems and researchers have the potential to drive synergistic advancements, leading to improved patient care and outcomes. Key Points
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several studies have shown that while the model can generate plausible-sounding citations, many of these are either partially incorrect or entirely fabricated, often combining real author names with nonexistent titles or invalid DOIs. 25-29”
From Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics (2025) · cites this paper as [27]
In the citing paper
“These hallucinated references have been documented across various disciplines, including medicine, education, and health sciences, raising widespread concerns about the reliability of AI-generated bibliographic content.”
From Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics (2025) · cites this paper as [27]
Citations
Cites
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
ChatGPT for healthcare services: An emerging stage for an innovative perspective
Javaid, M., Haleem, A., & Singh, R. P. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100105 · Cited by 429
Respiratory Syncytial Virus Vaccination during Pregnancy and Effects in Infants
Madhi, S. A., Polack, F. P., Piedra, P. A., Munoz, F. M., Trenholme, A. A., Simões, E. A. F., Swamy, G. K., Agrawal, S., Ahmed, K., August, A., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa1908380 · Cited by 418
Hoffman, M. K., Goudar, S. S., Kodkany, B. S., Metgud, M., Somannavar, M., Okitawutshu, J., Lokangaka, A., Tshefu, A., Bose, C. L., Mwapule, A., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32973-3 · Cited by 360
ChatGPT is shaping the future of medical writing but still requires human judgment
Kitamura, F. C. (2023) · Radiology · doi:10.1148/radiol.230171 · Cited by 317
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234
Pravastatin for early‐onset pre‐eclampsia: a randomised, blinded, placebo‐controlled trial
Ahmed, A. S., Williams, D. J., Cheed, V., Middleton, L. J., Ahmad, S., Wang, K., Vince, A., Hewett, P. W., Spencer, K., Khan, K. S., et al. (2019) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16013 · Cited by 155
Using ChatGPT for language editing in scientific articles
Kim, S.-G. (2023) · Maxillofacial Plastic and Reconstructive Surgery · doi:10.1186/s40902-023-00381-x · Cited by 153
A comparison of ChatGPT-generated articles with human-written articles
Ariyaratne, S., Iyengar, K. P., Nischal, N., Chitti Babu, N., & Botchu, R. (2023) · Skeletal Radiology · doi:10.1007/s00256-023-04340-5 · Cited by 148
Antenatal Dexamethasone for Early Preterm Birth in Low-Resource Countries
Collaborators, T. W. A. T. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2022398 · Cited by 128
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125
Shennan, A. H., Chandiramani, M., Bennett, P. R., David, A. L., Girling, J., Ridout, A. E., Seed, P. T., Simpson, N., Thornton, S., Tydeman, G., et al. (2019) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.09.040 · Cited by 122
Early gestational diabetes screening in obese women: a randomized controlled trial
Harper, L. M., Jauk, V., Longo, S., Biggio, J. R., Szychowski, J. M., & Tita, A. T. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.12.021 · Cited by 121
Physical examination–indicated cerclage in twin pregnancy: a randomized controlled trial
Roman, A., Zork, N. M., Haeri, S., Schoen, C. N., Saccone, G., Colihan, S., Zelig, C. M., Gimovsky, A. C., Seligman, N. S., Zullo, F., et al. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.06.047 · Cited by 111
Teigen, N. C., Sahasrabudhe, N., Doulaveris, G., Xie, X., Negassa, A., Bernstein, J., & Bernstein, P. S. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.10.009 · Cited by 105
Husain, S., Allotey, J., Drymoussi, Z., Wilks, M., Fernandez‐Felix, B., Whiley, A., Dodds, J., Thangaratinam, S., McCourt, C., Prosdocimi, E., et al. (2020) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.15675 · Cited by 96
Tuuli, M. G., Liu, J., Tita, A., Longo, S. A., Trudell, A., Carter, E. B., Shanks, A. L., Woolfolk, C., Caughey, A. B., Warren, D. K., et al. (2020) · JAMA · doi:10.1001/jama.2020.13361 · Cited by 94
Quotation accuracy in medical journal articles—a systematic review and meta-analysis
Jergas, H., & Baethge, C. (2015) · PeerJ · doi:10.7717/peerj.1364 · Cited by 88
Sangkomkamhang, U. S., Kongwattanakul, K., Kietpeerakool, C., Thinkhamrop, J., Wannasiri, P., Khunpradit, S., Thepsuthammarat, K., Jampathong, N., & Lumbiganon, P. (2019) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.15982 · Cited by 83
Chat Generative Pre-trained Transformer: why we should embrace this technology
Chavez, M. R., Butler, T. S., Rekawek, P., Heo, H., & Kinzler, W. L. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.010 · Cited by 83 · cited 5×
“While previous research has focused primarily on ChatGPT, 6 – 8 particularly in relation to arti fi cial hallucinations or confab-ulations, 9 – 15 this studyexpands upon the topic bycomparing the reliability of ChatGPT to other popular AI chatbots in providing citations to scienti fi c articles.”
BackgroundMethodInfluentialMogull, S. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184727 · Cited by 75
Accuracy of references in five leading medical journals
Siebers, R., & Holt, S. (2000) · The Lancet · doi:10.1016/s0140-6736(05)74090-3 · Cited by 71
Yu, S. C. H., Cheng, Y. K. Y., Tse, W. T., Sahota, D. S., Chung, M. Y., Wong, S. S. M., Chan, O. K., & Leung, T. Y. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.01.024 · Cited by 69
Akselsson, A., Lindgren, H. E., Georgsson, S., Pettersson, K., Steineck, G., Skokic, V., & Rådestad, I. J. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16104 · Cited by 68
Chen, M., Liu, X., You, Y., Wang, X., Li, T., Luo, H., Qu, H., & Xu, L. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003792 · Cited by 68
Beware of references when using ChatGPT as a source of information to write scientific articles
Sanchez-Ramos, L., Lin, L., & Romero, R. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.04.004 · Cited by 60
Self-citationKuppermann, M., Kaimal, A. J., Blat, C., Gonzalez, J., Thiet, M.-P., Bermingham, Y., Altshuler, A. L., Bryant, A. S., Bacchetti, P., & Grobman, W. A. (2020) · JAMA · doi:10.1001/jama.2020.5952 · Cited by 50
Beckmann, M. M., GIBBONS, K. S., Flenady, V., & Kumar, S. (2019) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16030 · Cited by 46
Wolf, H., Brok, J. S., Henriksen, T. B., Greisen, G. O., Salvig, J., Pryds, O. A., Hedegaard, M. S., Weber, T., Hegaard, H. K., Pinborg, A. B., et al. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16239 · Cited by 46
Elkind-Hirsch, K. E., Seidemann, E. L., & Harris, R. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100139 · Cited by 46
(1977) · PubMed · Cited by 44
Lewkowitz, A. K., López, J. D., Carter, E. B., DUCKHAM, H. E., Strickland, T., Macones, G. A., & Cahill, A. G. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100143 · Cited by 43
Ngai, F., Wong, P., Chung, K., Chau, P., & Hui, P. (2020) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.15862 · Cited by 42
Evaluation of an external fetal electrocardiogram monitoring system: a randomized controlled trial
Monson, M. A., Heuser, C. C., Einerson, B. D., Esplin, I., Snow, G. L., Varner, M. W., & Esplin, M. S. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.02.012 · Cited by 41
Slade, P., West, H., Thomson, G., Lane, S., Spiby, H., Edwards, R., Charles, J., Garrett, C., Flanagan, B., Treadwell, M., et al. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16163 · Cited by 39
Paidas, M. J., Tita, A. T. N., Macones, G. A., Saade, G. A., Ehrenkranz, R. A., Triche, E. W., Streisand, J. B., Lam, G. K., Magann, E. F., Lewis, D. F., et al. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.08.004 · Cited by 36
Outpatient Foley Catheter for Induction of Labor in Nulliparous Women
Ausbeck, E. B., Jauk, V. C., Xue, Y., Files, P., Kuper, S. G., Subramaniam, A., Casey, B. M., Szychowski, J. M., Harper, L. M., & Tita, A. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000004041 · Cited by 36
Science fact vs science fiction: A ChatGPT immunological review experiment gone awry
Wittmann, J. (2023) · Immunology Letters · doi:10.1016/j.imlet.2023.04.002 · Cited by 32
Shamshirsaz, A. A., Lee, T. C., Hair, A. B., Erfani, H., Espinoza, J., Shamshirsaz, A. A., Fox, K. A., Gandhi, M., Nassr, A. A., Abrams, S. A., et al. (2019) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.21871 · Cited by 30
Double-Balloon Device for 6 Compared With 12 Hours for Cervical Ripening
Bleicher, I., Dikopoltsev, E., Kadour-Ferro, E., Sammour, R. N., Gonen, R., Sagi, S., Eshel, A., Nussam, L., & Vitner, D. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003804 · Cited by 29
Dinis, J., Soto, E. E., Pedroza, C., Chauhan, S. P., Blackwell, S. C., & Sibai, B. M. (2019) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.12.001 · Cited by 25
A role for artificial intelligence chatbots in the writing of scientific articles
Vintzileos, A. M., Chavez, M. R., & Romero, R. J. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.040 · Cited by 25
Cephalic Elevation Device for Second-Stage Cesarean Delivery
Lassey, S. C., Little, S. E., Saadeh, M., Patton, N. M., Farber, M. K., Bateman, B. T., & Robinson, J. N. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003746 · Cited by 23
Moors, S., Bullens, L. M., Heimel, P. J. V. R., Dieleman, J. P., Kulik, W., Bakkeren, D. L., van den Heuvel, E. R., der Jagt, M. B. V. D. H., & Oei, S. G. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100102 · Cited by 23
Andreasen, L. A., Tabor, A., Nørgaard, L. N., Ringsted, C. V., Sandager, P., Rosthøj, S., & Tolsgaard, M. G. (2019) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.20362 · Cited by 20
Alternate Dosing Protocol for Magnesium Sulfate in Obese Women With Preeclampsia
Brookfield, K., Tuel, K., Rincon, M. P., Vinson, A., Caughey, A. B., & Carvalho, B. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000004137 · Cited by 20
Stone, J., Bianco, A., Monro, J., Overybey, J. R., Cadet, J., Choi, K. H., Pena, J., Robles, B. N., Mella, M. T., Matthews, K. C., et al. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.05.021 · Cited by 18
Reducing the Risk for Postpartum Depression in Adolescent Mothers: A Randomized Controlled Trial
PHIPPS, M. G., Ware, C. F., Stout, R. L., Raker, C., & Zlotnick, C. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000004003 · Cited by 18
Mireault, D., Loubert, C., Drolet, P. A., Tordjman, L., Godin, N., Richebé, P., & Zaphiratos, V. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003821 · Cited by 17
Foley Bulb Added to an Oral Misoprostol Induction Protocol
Adhikari, E. H., Nelson, D. B., McIntire, D. D., & Leveno, K. J. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000004123 · Cited by 16
Duffy, C. R., Garcia-So, J., Ajemian, B., Gyamfi‐Bannerman, C., & Han, Y. W. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100114 · Cited by 15
Tan, P. C., Abdussyukur, S., Lim, B., Win, S., & Omar, S. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16290 · Cited by 14
Hautakangas, T., Uotila, J. T., Huhtala, H., & Palomäki, O. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16343 · Cited by 14
Doulaveris, G., Vani, K., Saccone, G., Chauhan, S. P., & Berghella, V. (2021) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2021.100509 · Cited by 14
Accuracy of Reference Citations in Five Entomology Journals
Kristof, C. (1997) · American Entomologist · doi:10.1093/ae/43.4.246 · Cited by 13
Sherrell, H. C., Clifton, V. L., & Kumar, S. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.02.034 · Cited by 13
Ropivacaine and Ketorolac Wound Infusion for Post–Cesarean Delivery Analgesia
Barney, E. Z., Pedro, C. D., Gamez, B. H., Fuller, M., Dominguez, J. E., & Habib, A. S. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003601 · Cited by 12
Herman, H. G., Kleiner, I., Tairy, D., Gonen, N., Zvi, M. B., Kovo, M., Bar, J., & Weiner, E. (2020) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000003879 · Cited by 11
Choi, S., Kwak, D., Kil, K., Kim, S., Kwon, J., Kim, Y., Na, S., Bae, J., Cha, H.-H., Shim, J., et al. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16365 · Cited by 11
A randomised trial of caesarean wound coverage: exposed versus dressed
Tan, P. C., Rohani, E., Lim, M., Win, S., & Omar, S. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16228 · Cited by 10
A randomized controlled trial of liposomal bupivacaine for pain following obstetrical laceration
Dengler, K. L., Simpson, K. J., Strauchon, C., Shaddeau, A., Brooks, D. I., & Gruber, D. D. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100115 · Cited by 7
Cited by
Ng, J. Y. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101222 · Cited by 21
Zhao, M., Zhou, M., Han, Y., Song, X., Zhou, Y., & He, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-19370-3 · Cited by 21
Karaağaç, M., & Çarkıt, S. (2025) · Oncology · doi:10.1159/000546726 · Cited by 7
Safran, E., & Çalı, A. (2025) · Anatolian Current Medical Journal · doi:10.38053/acmj.1746227 · Cited by 4
Ng, J. Y. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1745928 · Cited by 8
“In health and medical research, this can manifest as misrepresented study findings, fictitious references, or misleading summaries of scientific literature (Walters and Wilder, 2023; Chen and Chen, 2023; Graf et al., 2024; Loh, 2023).”
Metadata
- Authors
- Graf, E. M., McKinney, J. A., Dye, A. B., Lin, L., & Sanchez-Ramos, L.
- Affiliations
- University of Florida (US)
- University of Arizona (US)
- Department of Obstetrics and Gynecology, University of Florida College of Medicine, Jacksonville, Florida
- Corresponding author
- McKinney, Jordan A.
- Venue
- American Journal of Perinatology
- Publisher
- Georg Thieme Verlag KG
- Published
- 2024-04-23
- Volume
- 41
- Issue
- 15
- Pages
- 2072-2081
- ISSN
- 0735-1631 (ISSN-L)
- 1098-8785
- OpenAlex
- W4395051595
- Semantic Scholar
- c0d8e76c4998bd86974e45254dbdeba1d1b3c74f
- Cited by
- 12
- References
- 61
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 2
- FWCI
- 1.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Perinatology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- AI chatbots, reference generation, citation accuracy, ChatGPT, Google Bard, digital object identifiers, responsible AI use, scientific writing
- MeSH terms
- Artificial Intelligence, Humans, Natural Language Processing, Obstetrics, Periodicals as Topic, Reproducibility of Results, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.