Cited by 87Artificial Intelligence in Healthcare and EducationHealth Literacy and Information AccessibilityReproductive Health and Technologies

The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations

Open access
Chervenak, Joseph, Lieman, Harry, Blanco-Breindel, Miranda, Jindal, Sangita·2023·Fertility and Sterility

Abstract

OBJECTIVE: To compare the responses of the large language model-based "ChatGPT" to reputable sources when given fertility-related clinical prompts. DESIGN: The "Feb 13" version of ChatGPT by OpenAI was tested against established sources relating to patient-oriented clinical information: 17 "frequently asked questions (FAQs)" about infertility on the Centers for Disease Control (CDC) Website, 2 validated fertility knowledge surveys, the Cardiff Fertility Knowledge Scale and the Fertility and Infertility Treatment Knowledge Score, as well as the American Society for Reproductive Medicine committee opinion "optimizing natural fertility." SETTING: Academic medical center. PATIENT(S): Online AI Chatbot. INTERVENTION(S): Frequently asked questions, survey questions and rephrased summary statements were entered as prompts in the chatbot over a 1-week period in February 2023. MAIN OUTCOME MEASURE(S): For FAQs from CDC: words/response, sentiment analysis polarity and objectivity, total factual statements, rate of statements that were incorrect, referenced a source, or noted the value of consulting providers. FOR FERTILITY KNOWLEDGE SURVEYS: Percentile according to published population data. FOR COMMITTEE OPINION: Whether response to conclusions rephrased as questions identified missing facts. RESULT(S): When administered the CDC's 17 infertility FAQ's, ChatGPT produced responses of similar length (207.8 ChatGPT vs. 181.0 CDC words/response), factual content (8.65 factual statements/response vs. 10.41), sentiment polarity (mean 0.11 vs. 0.11 on a scale of -1 (negative) to 1 (positive)), and subjectivity (mean 0.42 vs. 0.35 on a scale of 0 (objective) to 1 (subjective)). In total, 9 (6.12%) of 147 ChatGPT factual statements were categorized as incorrect, and only 1 (0.68%) statement cited a reference. ChatGPT would have been at the 87th percentile of Bunting's 2013 international cohort for the Cardiff Fertility Knowledge Scale and at the 95th percentile on the basis of Kudesia's 2017 cohort for the Fertility and Infertility Treatment Knowledge Score. ChatGPT reproduced the missing facts for all 7 summary statements from "optimizing natural fertility." CONCLUSION(S): A February 2023 version of "ChatGPT" demonstrates the ability of generative artificial intelligence to produce relevant, meaningful responses to fertility-related clinical queries comparable to established sources. Although performance may improve with medical domain-specific training, limitations such as the inability to reliably cite sources and the unpredictable possibility of fabricated information may limit its clinical use.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.fertnstert.2023.05.151

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts.

    “Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [21]

  • ChatGPT's limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.

    “Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [21]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Pallen M. Introducing the Internet. BMJ 1995;311:1422-4.
  2. Patel SB, Lam K, Liebrenz M. ChatGPT: friend or foe. Lancet Digit Health 2023;5:e102.
  3. Loria S, Keen P, Honnibal M, Yankovsky R, Karesh D, Dempsey E, et al. Text- blob: simplified text processing, Second. TextBlob Simpl. Text Process 2014; 3:2014.
  4. Agrawal A, Gans J, Goldfarb A. Power and prediction: the disruptive eco- nomics of artificial intelligence. Boston: Harvard Business Review Press; 2022.

Cited by

Sorted by citations

Authors
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S.
Affiliations
Albert Einstein College of Medicine (US)
Corresponding author
Chervenak, Joseph
Venue
Fertility and Sterility
Publisher
Elsevier BV
Published
2023-09
Volume
120
Issue
3
Pages
575-583
ISSN
0015-0282 (ISSN-L)
OpenAlex
W4377157938
Cited by
87
References
34
Citations by year
2026: 6 · 2025: 35 · 2024: 32 · 2023: 14
FWCI
3.52
Type
journal article
Language
en
Open access
Locations
  • Fertility and Sterility (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
  • Health Sciences › Medicine › Reproductive Medicine › Reproductive Health and Technologies
Keywords
ChatGPT, large language models, fertility counseling, infertility, natural fertility, AI chatbots, factual accuracy, sentiment analysis, referencing, patient-provider communication
MeSH terms
Artificial Intelligence, Counseling, Fertility, Humans, Infertility, Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review