Clinical Research With Large Language Models Generated Writing—Clinical Research with AI-assisted Writing (CRAW) Study
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
IMPORTANCE: The scientific community debates Generative Pre-trained Transformer (GPT)-3.5’s article quality, authorship merit, originality, and ethical use in scientific writing. OBJECTIVES: Assess GPT-3.5’s ability to craft the background section of critical care clinical research questions compared to medical researchers with H-indices of 22 and 13. DESIGN: Observational cross-sectional study. SETTING: Researchers from 20 countries from six continents evaluated the backgrounds. PARTICIPANTS: Researchers with a Scopus index greater than 1 were included. MAIN OUTCOMES AND MEASURES: In this study, we generated a background section of a critical care clinical research question on “acute kidney injury in sepsis” using three different methods: researcher with H-index greater than 20, researcher with H-index greater than 10, and GPT-3.5. The three background sections were presented in a blinded survey to researchers with an H-index range between 1 and 96. First, the researchers evaluated the main components of the background using a 5-point Likert scale. Second, they were asked to identify which background was written by humans only or with large language model-generated tools. RESULTS: A total of 80 researchers completed the survey. The median H-index was 3 (interquartile range, 1–7.25) and most (36%) researchers were from the Critical Care specialty. When compared with researchers with an H-index of 22 and 13, GPT-3.5 was marked high on the Likert scale ranking on main background components (median 4.5 vs. 3.82 vs. 3.6 vs. 4.5, respectively; p < 0.001). The sensitivity and specificity to detect researchers writing versus GPT-3.5 writing were poor, 22.4% and 57.6%, respectively. CONCLUSIONS AND RELEVANCE: GPT-3.5 could create background research content indistinguishable from the writing of a medical researcher. It was marked higher compared with medical researchers with an H-index of 22 and 13 in writing the background section of a critical care clinical research question.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In critical care, where decisions are time-sensitive and high-risk, the application of LLMs is rapidly expanding across clinical decision support, documentation, and education, including their use in generating scientific writing and research backgrounds (10, 11).”
From Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Likert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“Additionally, we compared the average scores on a Likert scale using t test (18, 19).”
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
Analyzing and Interpreting Data From Likert-Type Scales
Sullivan, G. M., & Artino, A. R. (2013) · Journal of Graduate Medical Education · doi:10.4300/jgme-5-4-18 · Cited by 3,044 · cited 1×
“Additionally, we compared the average scores on a Likert scale using t test (18, 19).”
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099 · cited 4×
“In this sense, Gao et al (14) evaluated the ability of humans to detect LLMs abstracts and found results similar to ours, with low sensitivity and specificity for researchers detecting them.”
InfluentialGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Can artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · cited 3×
“5-generated summaries reported by Salvagno et al (21) were believable but generic, sparse in details, and contained severe mistakes and false information.”
ChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571 · cited 1×
“Furthermore, it has the potential to become a powerful tool for scientific writing, including automated draft generation, article summarization, and language translation, which could help researchers write more efficiently and effectively (7).”
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023) · JAMA · doi:10.1001/jama.2023.1344 · Cited by 420 · cited 1×
“5, as well as their originality and the ethical and acceptable boundaries of using them in scientific writing (1, 12, 13).”
Cited by
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
Moulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Large Language Models in Critical Care Medicine: Scoping Review
Shi, T., Ma, J., Yu, Z., Xu, H., Yang, R., Xiong, M., Xiao, M., Li, Y., Zhao, H., & Kong, G. (2025) · JMIR Medical Informatics · doi:10.2196/76326 · Cited by 21
Assessing the Capability of Large Language Model Chatbots in Generating Plain Language Summaries
Mondal, H., Gupta, G., Sarangi, P. K., Sharma, S., Choudhary, P. K., Juhi, A., Kumari, A., & Mondal, S. (2025) · Cureus · doi:10.7759/cureus.80976 · Cited by 14
Seifi, A., & Seyfi, A. (2026) · Critical Care Explorations · doi:10.1097/cce.0000000000001474 · Cited by 1
Metadata
- Authors
- Huespe, I. A., Echeverri, J., Khalid, A., Carboni Bisso, I., Musso, C. G., Surani, S., Bansal, V., & Kashyap, R.
- Venue
- Critical Care Explorations
- Published
- 2023-10
- Volume
- 5
- Issue
- 10
- Pages
- e0975
- ISSN
- 2639-8028 (ISSN-L)
- OpenAlex
- W4387249771
- Cited by
- 27
- References
- 16
- FWCI
- 0.93
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Clinical Research With Large Language Models Generated Writing—Clinical Research with AI-assisted Writing (CRAW) Study. https://app.cochat.ai/papers/10.1097/cce.0000000000000975
- The paper
- Huespe, I. A., Echeverri, J., Khalid, A., Carboni Bisso, I., Musso, C. G., Surani, S., Bansal, V., & Kashyap, R. (2023). Clinical Research With Large Language Models Generated Writing—Clinical Research with AI-assisted Writing (CRAW) Study. Critical Care Explorations. https://doi.org/10.1097/cce.0000000000000975
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.