Analysing the impact of ChatGPT in research
Open accessAbstract
Large Language Models (LLMs) are a type of machine learning that handles a wide range of Natural Language Processing (NLP) scenarios. Recently, in December 2022, a company called OpenAI released ChatGPT, a tool that, within a few months, became the most representative example of LLMs, automatically generating unique and coherent text on many topics, summarising and rewriting it, or even translating it to other languages. ChatGPT originated some controversy in academia since students can generate unique text for writing assessments being sometimes extremely difficult to distinguish whether it comes from ChatGPT or a person. In research, some journals specifically banned ChatGPT in scientific papers. However, when used correctly, it becomes a powerful tool to rewrite, for instance, scientific papers and, thus, deliver researchers’ messages in a better way. In this paper, we conduct an empirical study of the impact of ChatGPT in research. We downloaded the abstract of over 45,000 papers from over 300 journals from Dec 2022 and Feb 2023 belonging to different research editorials. We use four of the most known ChatGPT detection tools and conclude that ChatGPT played a role in around 10% of the papers published in every editorial, showing that authors from different fields have rapidly adopted such a tool in their research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some studies have reported differences in estimated LLM usage between English- and non-English-speaking countries, academic fields, and publishing venues.
“Some studies have reported differences in estimated LLM usage between English- and non-English–speaking countries (18–20), academic fields (17), and publishing venues.”
From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [20]
Many scholars have embraced the use of the ChatGPT application to structure and assist in writing their manuscripts.
“Many scholars have embraced the use of the ChatGPT application to structure and assist in writing their manuscripts (Picazo-Sanchez, and Ortiz-Martin, 2024).”
From Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review (2024) · cites this paper as Picazo-Sanchez, and Ortiz-Martin, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,089 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Chomsky, N. (1957) · doi:10.1515/9783112316009 · Cited by 7,885 · see citing works
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · see citing works
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939 · see citing works
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · see citing works
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1× · see citing works
“The term Language Model (LM) refers to systems trained on string prediction tasks [1].”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206148 · Cited by 807 · see citing works
The fickle P value generates irreproducible results
Halsey, L. G., Curran‐Everett, D., Vowler, S. L., & Drummond, G. B. (2015) · Nature Methods · doi:10.1038/nmeth.3288 · Cited by 753 · see citing works
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611 · see citing works
Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?
O’Connor, S., & ChatGPT (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103537 · Cited by 585 · see citing works
The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two?
Lehmann, E. L. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476404 · Cited by 384 · see citing works
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
Distribution of content words and phrases in text and language modelling
Katz, S. M. (1996) · Natural Language Engineering · doi:10.1017/s1351324996001246 · Cited by 259 · see citing works
ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE PART I
Neyman, J., & PEARSON, E. S. (1928) · Biometrika · doi:10.1093/biomet/20a.1-2.175 · Cited by 259 · see citing works
Church, K., & Gale, W. A. (1995) · Natural Language Engineering · doi:10.1017/s1351324900000139 · Cited by 215 · see citing works
Nationality Classification Using Name Embeddings
Ye, J., Han, S., Hu, Y., et al. (2017) · doi:10.1145/3132847.3133008 · Cited by 114 · see citing works
Link prediction in co-authorship networks based on hybrid content similarity metric
Chuan, P. M., Sơn, L. H., Ali, M., et al. (2017) · Applied Intelligence · doi:10.1007/s10489-017-1086-x · Cited by 103 · see citing works
O’Connor, S. (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2023.103572 · Cited by 40 · see citing works
How language-generation AIs could transform science
Noorden, R. V. (2022) · Nature · doi:10.1038/d41586-022-01191-3 · Cited by 23 · see citing works
Flagged first, then by citations
Research works printed but not linked: 10
- Lepikhin D, Lee H, Xu Y, Chen D, Firat O, Huang Y, Krikun M, Shazeer N, Chen Z (2021) Gshard: Scaling giant models with conditional computation and automatic sharding. In: International conference on learning representations
- De Saussure F (1989) Cours de Linguistique Générale vol. 1. Otto Harrassowitz Verlag
- Guo B, Zhang X, Wang Z, Jiang M, Nie J, Ding Y, Yue J, Wu Y (2023) How close is ChatGPT to human experts? comparison cor- pus, evaluation, and detection. In: Symposium on large language models, colocated with the international joint conference on artifi- cial intelligence
- The Lancet: Information for Authors. https://www.thelancet. com/pb/assets/raw/Lancet/authors/tl-info-for-authors- 1676565160037.pdf
- Elsevier: To Err is Not Human: The Dangers of AI-assisted Academic Writing. https://scientific-publishing.webshop.elsevier. com/research-process/the-dangers-of-ai-assisted-academic- writing/
- Elsevier: Publishing Ethics -The Use of AI and AI-assisted Tech- nologies in Scientific Writing. https://www.elsevier.com/about/ policies/publishing-ethics#Authors
- Elsevier: The use of AI and AI-assisted writing technolo- gies in scientific writing. https://www.elsevier.com/about/ policies/publishing-ethics/the-use-of-ai-and-ai-assisted-writing- technologies-in-scientific-writing 37. Science: Editorial Policies. https://www.science.org/content/page/ science-journals-editorial-policies 38. Springer: Authorship Principles. https://www.springer.com/us/ editorial-policies/authorship-principles 39. Elsevier: The use of generative AI and AI-assisted technolo- gies in writing for Elsevier. https://www.elsevier.com/about/ policies-and-standards/the-use-of-generative-ai-and-ai-assisted- technologies-in-writing-for-elsevier
- IEEE: Submission and Peer Review Policies. https://www.elsevier. com/about/policies-and-standards/publishing-ethics# 41. MDPI: Research and Publication Ethics. https://www.mdpi.com/ ethics
- Science: Change to policy on the use of generative AI and large language models. https://www.science.org/content/blog-post/ change-policy-use-generative-ai-and-large-language-models 43. Springer Nature Group: Springer Nature introduces Curie, its AI-powered scientific writing assistant. https://group. springernature.com/la/group/media/press-releases/ai-powered- scientific-writing-assitant-launched/26176230
- Lancet T The use of AI and AI-assisted technologies in scientific writing. https://www.thelancet.com/publishing-excellence
Cited by
Sorted by citations
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213 · see citing works
“Some studies have reported differences in estimated LLM usage between English-and non-English-speaking countries (Cheng et al., 2024; Liu and Bu, 2024; Picazo-Sanchez and Ortiz-Martin, 2024), academic fields (Akram, 2024), and publishing venues.”
Current trends and future implications in the utilization of ChatGPT in nursing: a rapid review
Kleib, M., Darko, E. M., Akingbade, O., Kennedy, M., Majekodunmi, P., Nickel, E., & Vogelsang, L. (2024) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2024.100252 · Cited by 40 · see citing works
Bao, H., Sun, M., & Teplitskiy, M. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00368 · Cited by 7 · see citing works
Kousha, K., & Thelwall, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05601-5 · Cited by 10 · see citing works
ChatGPT as speechwriter for the French presidents
Labbé, D., Labbé, C., & Savoy, J. (2025) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf032 · Cited by 3 · see citing works
Picazo-Sanchez, P., & Ortiz-Martin, L. (2026) · Data · doi:10.3390/data11050122 · Cited by 1 · see citing works
“Only after the non-existence was confirmed bibliographically did we randomly select a few paragraphs from the paper and checked them using some of the https://doi.org/10.3390/data11050122 most well-known AI content detectors available, which is similar to previous research in the field [9].”
InfluentialSelf-citationLiu, J., He, Y., Zheng, Z., Bu, Y., & Ni, C. (2025) · arXiv · doi:10.48550/arxiv.2511.15872 · Cited by 1 · see citing works
- Authors
- Picazo-Sanchez, P., & Ortiz-Martin, L.
- Venue
- Applied Intelligence
- Published
- 2024-03
- Volume
- 54
- Issue
- 5
- Pages
- 4172-4188
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W4393053782
- Cited by
- 46
- References
- 22
- Citations by year
- 2026: 13 · 2025: 25 · 2024: 8
- FWCI
- 19.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Applied Intelligence (publishedVersion, cc-by)
- Funders
- Halmstad University
- Topics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, academic writing, scientific abstracts, research impact, text rewriting, AI-generated text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
