CoChat
Sign Up
PreprintCited by 6Academic integrity and plagiarismAuthorship Attribution and ProfilingArtificial Intelligence in Education

The DOI matches its DataCite record.

Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool

Open access
Akram, Arslan·2024·arXiv
DOI 10.48550/arxiv.2403.13812arXiv 2403.13812
Open this paper in CoChat — free accountOpen PDF

Abstract

Many people are interested in ChatGPT since it has become a prominent AIGC model that provides high-quality responses in various contexts, such as software development and maintenance. Misuse of ChatGPT might cause significant issues, particularly in public safety and education, despite its immense potential. The majority of researchers choose to publish their work on Arxiv. The effectiveness and originality of future work depend on the ability to detect AI components in such contributions. To address this need, this study will analyze a method that can see purposely manufactured content that academic organizations use to post on Arxiv. For this study, a dataset was created using physics, mathematics, and computer science articles. Using the newly built dataset, the following step is to put originality.ai through its paces. The statistical analysis shows that Originality.ai is very accurate, with a rate of 98%.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.13812

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some studies have reported differences in estimated LLM usage between English- and non-English-speaking countries, academic fields, and publishing venues.

    “Some studies have reported differences in estimated LLM usage between English- and non-English–speaking countries (18–20), academic fields (17), and publishing venues.”

    From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [17]

    The Results and Discussions section says the paper analyzed AI-content scores across computer science, mathematics, and physics, supporting its relevance to academic fields. The available passage does not report a difference in estimated usage between those fields. The country and publishing-venue parts of the citing sentence are not attached to this source's citation. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 21

  1. Baki, S., Verma, R., Mukherjee, A., & Gnawali, O. (2017, April). Scaling and effectiveness of email masquerade attacks: Exploiting natural language generation. In Proceedings of the 2017 ACM on Asia conference on computer and communications security (pp. 469-482). · DOI
  2. Shu, K., Wang, S., Lee, D., & Liu, H. (2020). Mining disinformation and fake news: Concepts, methods, and recent advancements. In Disinformation, misinformation, and fake news in social media: Emerging research challenges and opportunities (pp. 1-19). Cham: Springer International Publishing. · DOI
  3. Stiff, H., & Johansson, F. (2022). Detecting computer- generated disinformation. International Journal of Data Science and Analytics, 13(4), 363-383. · DOI · Paper page
  4. Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). Ethics in Science and Environmental Politics, 21, 17-23. · DOI · Paper page
  5. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
  6. Fortuna, P., & Nunes, S. (2018). A survey on automatic detection of hate speech in text. Acm Computing Surveys (Csur), 51(4), 1-30. · DOI · Paper page
  7. "Monthly Submissions." Accessed: Feb. 09, 2024. arXiv [Online].
  8. Kashnitsky, Y., Herrmannova, D., De Waard, A., Tsatsaronis, G., Fennell, C. C., & Labbé, C. (2022, October). Overview of the DAGPap22 shared task on detecting automatically generated scientific papers. In Proceedings of the Third Workshop on Scholarly Document Processing (pp. 210-213).
  9. Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., ... & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 1-39. · DOI · Paper page
  10. Burdisso, S. G., Errecalde, M. L., & Montes y Gómez, M. (2021). Using text classification to estimate the depression level of reddit users. Journal of Computer Science & Technology, 21.
  11. Wu, Y., Guan, S., & Wang, G. (2023). Drug effect deep learner based on graphical convolutional network. In Machine learning and deep learning in computational toxicology (pp. 83-140). Cham: Springer International Publishing. · DOI
  12. Akram, A., Rashid, J., Jaffar, M. A., Faheem, M., & Amin, R. U. (2023). Segmentation and classification of skin lesions using hybrid deep learning method in the Internet of Medical Things. Skin Research and Technology, 29(11), e13524. · DOI
  13. Akram, A., Rashid, J., Jaffar, A., Hajjej, F., Iqbal, W., & Sarwar, N. (2024). Weber law based approach for multi-class image forgery detection. Computers, Materials, & Continua, 78(1), 145. · DOI
  14. Akram, A., Ramzan, S., Rasool, A., Jaffar, A., Furqan, U., & Javed, W. (2022). Image splicing detection using discriminative robust local binary pattern and support vector machine. World Journal of Engineering, 19(4), 459-466. · DOI
  15. Akram, A., Khan, I., Rashid, J., Saddique, M., Idrees, M., Ghadi, Y., & Algarni, A. (2024). Enhanced steganalysis for color images using curvelet features and support vector machine. Computers, Materials, & Continua, 78(1), 1311. · DOI
  16. Grimaldi, G., & Ehrler, B. (2023). AI et al.: Machines are about to change scientific publishing forever. ACS energy letters, 8(1), 878-880. · DOI
  17. Mohammed, O., Sahib, T. M., Akhtom, D. A., Hayder, I. M., Salisu, S., & Shahid, M. (2023). ChatGPT Evaluation: Can It Replace Grammarly and Quillbot Tools?. British Journal of Applied Linguistics, 3(2), 34-46. · DOI · Paper page
  18. Chaka, C. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning & Teaching, 6(2), 94-104. · DOI · Paper page
  19. Müller, M., Salathé, M., & Kummervold, P. E. (2023). Covid- twitter-bert: A natural language processing model to analyse covid-19 content on twitter. Frontiers in artificial intelligence, 6, 1023281. · DOI
  20. "AI Content Detector Accuracy Review + Open Source Dataset and Research Tool -Originality.AI." Accessed: Feb. 09, 2024. [Online].
  21. Akram, A. (2023). An Empirical Study of AI-Generated Text Detection Tools. Adv Mach Lear Art Inte, 4(2), 44-55.

Cited by

Sorted by citations

Metadata

Authors
Akram, A.
Affiliations
Superior University (PK)
Corresponding author
Akram, Arslan
ORCID
Akram, Arslan
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
01
Issue
06
Pages
01-06
ISSN
2769-545X (ISSN-L)
DOI
10.48550/arxiv.2403.13812
arXiv
2403.13812
OpenAlex
W4393110942
Cited by
6
References
0
Citations by year
2025: 2 · 2024: 4
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Advances in Machine Learning & Artificial Intelligence (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
AI-generated text detection, ChatGPT, scholarly publishing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review