Cited by 59Artificial Intelligence in Healthcare and EducationAcademic Writing and PublishingMeta-analysis and systematic reviews

Can ChatGPT assist authors with abstract writing in medical journals? Evaluating the quality of scientific abstracts generated by ChatGPT and original abstracts

Open access

Abstract

Introduction: ChatGPT, a sophisticated large language model (LLM), has garnered widespread attention for its ability to mimic human-like communication. As recent studies indicate a potential supportive role of ChatGPT in academic writing, we assessed the LLM’s capacity to generate accurate and comprehensive scientific abstracts from published Randomised Controlled Trial (RCT) data, focusing on the adherence to the Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) statement, in comparison to the original authors’ abstracts. Methodology: RCTs, identified in a PubMed/MEDLINE search post-September 2021 across various medical disciplines, were subjected to abstract generation via ChatGPT versions 3.5 and 4, following the guidelines of the respective journals. The overall quality score (OQS) of each abstract was determined by the total number of adequately reported components from the 18-item CONSORT-A checklist. Additional outcome measures included percent adherence to each CONOSORT-A item, readability, hallucination rate, and regression analysis of reporting quality determinants. Results: Original abstracts achieved a mean OQS of 11.89 (95% CI: 11.23–12.54), outperforming GPT 3.5 (7.89; 95% CI: 7.32–8.46) and GPT 4 (5.18; 95% CI: 4.64–5.71). Compared to GPT 3.5 and 4 outputs, original abstracts were more adherent with 10 and 14 CONSORT-A items, respectively. In blind assessments, GPT 3.5-generated abstracts were deemed most readable in 62.22% of cases which was significantly greater than the original (31.11%; P = 0.003) and GPT 4-generated (6.67%; P<0.001) abstracts. Moreover, ChatGPT 3.5 exhibited a hallucination rate of 0.03 items per abstract compared to 1.13 by GPT 4. No determinants for improved reporting quality were identified for GPT-generated abstracts. Conclusions: While ChatGPT could generate more readable abstracts, their overall quality was inferior to the original abstracts. Yet, its proficiency to concisely relay key information with minimal error holds promise for medical research and warrants further investigations to fully ascertain the LLM’s applicability in this domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0297701

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior studies have mostly discussed potential applications and ethical implications of LLMs for scientific writing in medicine overall and, when performance was measured, analyzed often only one (typically proprietary) LLM.

    “Prior studies have mostly discussed potential applications and ethical implications of LLMs for scientific writing in medicine overall and – when performance was measured – analyzed often only one (typically proprietary) LLM.12–17”

    From Open-source versus proprietary large language models for academic writing in medicine: A controlled evaluation of DeepSeek-R1 and ChatGPT-o1 in neuroradiology (2026) · cites this paper as [16]

    The Introduction (page not available) discusses ChatGPT’s potential for scientific writing, and the Discussion addresses risks including bias, fabricated content and plagiarism. The paper also measures abstract-writing performance, but its Discussion explicitly says it evaluated both GPT 3.5 and GPT 4; it therefore does not support the suggestion that this study analyzed only one model. It does not establish what most prior studies did. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. OpenAI, 2023, GPT-4

Cited by

Sorted by citations

Authors
Hwang, T., Aggarwal, N., Khan, P. Z., Roberts, T., Mahmood, A., Griffiths, M. M., Parsons, N., & Khan, S.
Affiliations
  • University Hospitals Coventry and Warwickshire NHS Trust (GB)
  • University of Warwick (GB)
  • Royal Devon and Exeter Hospital (GB)
Corresponding author
Hwang, Taesoon
Venue
PLOS ONE
Published
2024-02-14
Volume
19
Issue
2
Pages
e0297701
ISSN
1932-6203 (ISSN-L)
OpenAlex
W4391811744
Cited by
59
References
20
Citations by year
2026: 22 · 2025: 22 · 2024: 15
FWCI
4.76
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (publishedVersion)
  • Warwick Research Archive Portal (University of Warwick) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Royal Devon and Exeter Research Repository (Royal Devon and Exeter) (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
ChatGPT, large language models, scientific abstracts, abstract writing, reporting quality, randomized controlled trials, hallucinations, readability, medical journal publishing, reporting guidelines adherence, publication ethics, plagiarism detection
MeSH terms
Hallucinations, Humans, Periodicals as Topic, Publications, Reference Standards, Writing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review