Towards AI-Generated Essay Classification Using Numerical Text Representation
Open accessAbstract
The detection of essays written by AI compared to those authored by students is increasingly becoming a significant issue in educational settings. This research examines various numerical text representation techniques to improve the classification of these essays. Utilizing a diverse dataset, we undertook several preprocessing steps, including data cleaning, tokenization, and lemmatization. Our system analyzes different text representation methods such as Bag of Words, TF-IDF, and fastText embeddings in conjunction with multiple classifiers. Our experiments showed that TF-IDF weights paired with logistic regression reached the highest accuracy of 99.82%. Methods like Bag of Words, TF-IDF, and fastText embeddings achieved accuracies exceeding 96.50% across all tested classifiers. Sentence embeddings, including MiniLM and distilBERT, yielded accuracies from 93.78% to 96.63%, indicating room for further refinement. Conversely, pre-trained fastText embeddings showed reduced performance, with a lowest accuracy of 89.88% in logistic regression. Remarkably, the XGBoost classifier delivered the highest minimum accuracy of 96.24%. Specificity and precision were above 99% for most methods, showcasing high capability in differentiating between student-created and AI-generated texts. This study underscores the vital role of choosing dataset-specific text representations to boost classification accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bag of Words and fastText consistently produced accuracies above 96.50%.
“Additionally, methods like Bag of Words and fastText consistently produced accuracies above 96.50%.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
The Conclusions state that Bag of Words and fastText models trained on the study’s dataset exceeded 96.50% accuracy with all classifiers. But the claim leaves fastText unqualified: the Results report that a pretrained fastText model fell to 89.88% with logistic regression, so fastText did not consistently exceed the threshold across all versions tested. (LLM, conf=0.89)
Experimental — accuracy not yet measuredAI detection tools remain unreliable for high-stakes academic decisions.
“In summary, while AI detection tools have shown promise, they remain unreliable for high-stakes academic decisions [22].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
The Related Works section reports that existing detectors can be inaccurate and unreliable, and the Conclusions note limitations of general-purpose classifiers. But the paper does not assess their suitability for high-stakes academic decisions; its own essay classifiers achieved high accuracy. No page numbers were supplied in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredMany current detectors rely heavily on outdated linguistic heuristics and were trained on earlier-generation models, making them less effective at identifying advanced AI-generated content.
“Many current detectors rely heavily on outdated linguistic heuristics and were trained on earlier-generation models, making them less effective at identifying advanced AI-generated content.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
Section 2 (Related Works; page not provided) says early detectors used linguistic or statistical patterns and became less precise as language models evolved. It does not establish that many current detectors rely heavily on outdated heuristics or were trained on earlier-generation models. (LLM, conf=0.86)
Experimental — accuracy not yet measuredWhite-box methods offer full access to a language model's internal mechanisms, supporting real-time behavior tracking.
“These white-box methods offer full access to a language model's internal mechanisms, supporting real-time behavior tracking.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
The Related Works section says white-box detection provides full access to the target language model and allows its behavior to be monitored and controlled. It does not specify real-time behavior tracking, so that qualifier is not supported by the passage. (LLM, conf=0.84)
Experimental — accuracy not yet measuredTF-IDF paired with logistic regression achieved the highest overall accuracy of 99.82%.
“Among these, TF-IDF paired with logistic regression achieved the highest overall accuracy of 99.82%.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
Researchers have begun exploring white-box detection methods, which involve embedding watermarking techniques directly into AI-generated text.
“In response, researchers have begun exploring white-box detection methods, which involve embedding watermarking techniques directly into AI-generated text [22].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
Sentence embeddings such as MiniLM and distilBERT yielded slightly lower accuracies (93.78-96.63%), suggesting further room for improvement.
“Meanwhile, sentence embeddings such as MiniLM and distilBERT yielded slightly lower accuracies (93.78-96.63%),suggesting further room for improvement [22].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
Inference-time watermarking embeds imperceptible yet verifiable patterns during the generation process, enabling systems to trace authorship.
“For instance, inference-time watermarking embeds imperceptible yet verifiable patterns during the generation process, enabling systems to trace authorship [22].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
The effectiveness of white-box methods depends heavily on whether institutions or commercial platforms adopt standardized watermarking protocols, which currently remains inconsistent.
“However, their effectiveness depends heavily on whether institutions or commercial platforms adopt standardized watermarking protocols, which currently remains inconsistent.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [22]
The Related Works section says white-box detection can use watermarks and describes properties effective watermarks require. It does not address whether institutions or commercial platforms adopt standardized protocols, whether adoption is inconsistent, or how adoption affects effectiveness (searched: standardized protocols, adoption, deployment). (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works
Advances in natural language processing
Hirschberg, J., & Manning, C. D. (2015) · Science · doi:10.1126/science.aaa8685 · Cited by 1,712 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
ENAI Recommendations on the ethical use of Artificial Intelligence in Education
Foltynek, T., Bjelobaba, S., Glendinning, I., Khan, Z. R., Santos, R., Pavletic, P., & Kravjar, J. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00133-4 · Cited by 180 · see citing works
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161 · see citing works
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · see citing works
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106 · see citing works
BERT and fastText Embeddings for Automatic Detection of Toxic Speech
d'Sa, A. G., Illina, I., & Fohr, D. (2020) · International Multi-Conference on Organization of Knowledge and Advanced Technologies (OCTA) · doi:10.1109/octa49274.2020.9151853 · Cited by 102 · see citing works
Machine learning in incident categorization automation
Silva, S., Pereira, R., & Ribeiro, R. (2018) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti.2018.8399244 · Cited by 32 · see citing works
Classification of Phishing Email Using Word Embedding and Machine Learning Techniques
Somesha, M., & Pais, A. R. (2022) · Journal of Cyber Security and Mobility · doi:10.13052/jcsm2245-1439.1131 · Cited by 27 · see citing works
Detecting textual adversarial examples through text modification on text classification systems
Kwon, H., & Lee, S.-H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03313-w · Cited by 14 · see citing works
A Framework for Detecting AI-Generated Text in Research Publications
Sarzaeim, P., Doshi, A. M., & Mahmoud, Q. H. (2023) · Proceedings of the International Conference on Advanced Technologies · doi:10.58190/icat.2023.28 · Cited by 11 · see citing works
Semantic Representation and Inference for NLPPreprint
Wang, D. (2020) · arXiv (Cornell University) · doi:10.13140/rg.2.2.34755.94243 · Cited by 3 · see citing works
Detecting Textual Backdoor Attacks via Class Difference for Text Classification System
Kwon, H., & Jun, L. E. E. (2024) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2023edp7160 · Cited by 2 · see citing works
Inżynieria języka dla systemu Thetos
Suszczańska, N., & Szmal, P. (2013) · doi:10.21936/si2011_v32.n4a.217 · Cited by 1 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 9
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv 2019, arXiv:1907.11692. · arXiv
- Tabassum, A.; Patil, R.R. A Survey on Text Pre-Processing & Feature Extraction Techniques in Natural Language Processing. Int. Res. J. Eng. Technol. (IRJET) 2020, 7, 4864-4867.
- Joulin, A.; Grave, E.; Bojanowski, P.; Douze, M.; Jégou, H.; Mikolov, T. FastText.zip: Compressing text classification models. arXiv 2016, arXiv:1612.03651. · arXiv
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Polosukhin, I. Attention is all you need. arXiv 2017, arXiv:1706.03762. · arXiv
- OpenAI. GPT-4 Technical Report. arXiv 2023, arXiv:2303.08774. · arXiv
- Shyam, P.; Sastry, G.; Askell, A. Language models are few-shot learners. arXiv 2020, arXiv:2005.14165. · arXiv
- Ziegler, D.M.; Stiennon, N.; Wu, J.; Brown, T.B.; Radford, A.; Amodei, D.; Irving, G. Fine-tuning language models from human preferences. arXiv 2019, arXiv:1909.08593. · arXiv
- Naveed, H.; Khan, A.U.; Qiu, S.; Saqib, M.; Anwar, S.; Usman, M.; Mian, A. A Comprehensive Overview of Large Language Models. arXiv 2023, arXiv:2307.06435. · arXiv
- Sanh, V.; Debut, L.; Chaumond, J.; Wolf, T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv 2019, arXiv:1910.01108 · arXiv
Cited by
Sorted by citations
Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education
Deep, P. D., Edgington, W. D., Ghosh, N., & Rahaman, M. S. (2025) · Information · doi:10.3390/info16100905 · Cited by 26 · see citing works
Ragab, M., Ashary, E. B., Kateb, F., Hakeem, A., Mosli, R., Albogami, N., & Nooh, S. A. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.04.071 · Cited by 7 · see citing works
Wang, S. (2025) · Applied Sciences · doi:10.3390/app15031088 · Cited by 2 · see citing works
Vaidya, S., Rathod, R. G., Saini, J. R., & Tamhankar, P. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-31216-7_9
- Authors
- Krawczyk, N., Probierz, B., & Kozak, J.
- Affiliations
- University of Economics in Katowice (PL)
- Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland
- Corresponding author
- Probierz, Barbara
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-10-26
- Volume
- 14
- Issue
- 21
- Pages
- 9795
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4403810633
- Cited by
- 4
- References
- 19
- Citations by year
- 2026: 1 · 2025: 3
- FWCI
- 0.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- AI-generated text detection, TF-IDF, bag-of-words, FastText embeddings, sentence embeddings, MiniLM, DistilBERT, logistic regression, XGBoost, text preprocessing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
