Admissions in the age of AI: detecting AI-generated application materials in higher education
Open accessAbstract
Recent advances in Artificial Intelligence (AI), such as the development of large language models like ChatGPT, have blurred the boundaries between human and AI-generated text. This has led to a pressing need for tools that can determine whether text has been created or revised using AI. A general and universally effective detection model would be extremely useful, but appears to be beyond the reach of current technology and detection methods. The research described in this study adopts a domain and task specific approach and shows that specialized detection models can attain high accuracy. The study focuses on the higher education graduate admissions process, with the specific goal of identifying AI-generated and AI-revised Letters of Recommendation (LORs) and Statements of Intent (SOIs). Detecting such application materials is essential to ensure that applicants are evaluated on their true merits and abilities, and to foster an equitable and trustworthy admissions process. Our research is based on 3755 LORs and 1973 SOIs extracted from the application records of Fordham University's Master's programs in Computer Science and Data Science. To facilitate the construction and evaluation of detection models, we generated AI counterparts for each LOR and SOI using the GPT-3.5 Turbo API. The prompts for AI-generation text were derived from the admission data of the respective applicants, and the AI-revised LORs and SOIs were generated directly from the human-authored versions. We also utilize an open-access GPT-wiki-intro dataset to further validate our hypothesis regarding the feasibility of constructing domain-specific AI content detectors. Our experiments yield promising results in developing classifiers tailored to a specific domain when provided with sufficient training samples. Additionally, we present a comparative analysis of the word frequency and statistical characteristics of the text, which provides convincing evidence that ChatGPT employs distinctive vocabulary and paragraph structure compared to human-authored text. The code for this study is available on GitHub, and the models can be executed on user-provided data via an interactive web interface.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many universities have implemented AI detection tools such as Turnitin AI, GPTZero, Copyleaks, and ZeroGPT to identify AI-generated content in student work.
“Many universities have implemented AI detection tools like Turnitin AI, GPTZero, Copyleaks, and ZeroGPT to identify AI-generated content in student work.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Related work section says Turnitin has expanded from plagiarism detection to AI-generated text, supporting Turnitin as an example of such a tool. It does not say that many universities have implemented detection tools or identify GPTZero, Copyleaks, or ZeroGPT as tools they use. The Conclusion discusses how higher education institutions could use detection tools for admissions, not widespread implementation for student work. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThis exploitation concerns scholarly integrity and the legitimacy of assignments submitted by students, especially where detection tools are not yet proven highly reliable or limited to a specific domain.
“This exploitation concerns scholarly integrity and the legitimacy of assignments submitted by students, especially where detection tools are not yet proven highly reliable or limited to a specific domain [8,25].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction links AI-generated application materials to academic integrity and says accurate universal AI-content detectors are currently unattainable. But this study concerns graduate-admissions materials—letters of recommendation and statements of intent—not the legitimacy of assignments submitted by students. It also reports that domain-specific detectors can be highly accurate. (LLM, conf=0.86)
Experimental — accuracy not yet measuredIncreased uses of AI in higher education are leading to a reevaluation of core assessment approaches.
“Increased uses of AI in higher education are leading to a reevaluation of core assessment approaches [4,8,25].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction (page unavailable) says increasing use of AI writing tools challenges how admissions committees evaluate applicants’ work. The Conclusion recommends AI detection to protect admissions evaluations, and Related work discusses redesigning assignments. These passages support changes to specific assessment practices, but not a general reevaluation of core assessment approaches across higher education. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTraditional forms, including untimed take-home essays or standardized problem sets, grow increasingly vulnerable to exploitation by AI products like ChatGPT.
“Traditional forms, including untimed take-home essays or standardized problem sets, grow increasingly vulnerable to exploitation by AI products of the sort of ChatGPT.”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction (page unavailable) says generative AI challenges admissions committees evaluating applicants’ essays and other materials, supporting the concern about AI-assisted written work. The paper does not address untimed take-home essays as a specific assessment format or standardized problem sets; its study focuses on admissions materials. (LLM, conf=0.84)
Experimental — accuracy not yet measuredEven transformer-based detectors such as DistilBERT experience a notable decline in performance when LLMs are prompted or fine-tuned to mimic a student's writing style.
“Even transformer-based detectors such as DistilBERT experience a notable decline in performance when LLMs are prompted or fine-tuned to mimic a student's writing style [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Methods identify DistilBERT as a transformer-based detector, and the Conclusion says refined prompts may thwart detection models. But the paper did not test LLMs prompted or fine-tuned to mimic a student's writing style, so it does not establish the claimed performance decline under that condition. In Results (Models trained on SOIs and LORs), DistilBERT achieved over 99% accuracy on AI-revised application documents. (LLM, conf=0.84)
Experimental — accuracy not yet measured
These stylistic adaptations increase the likelihood of false negatives, suggesting that detection systems relying solely on syntactic or stylistic markers may be inadequate for identifying AI-generated content that closely resembles human writing.
“These stylistic adaptations increase the likelihood of false negatives, suggesting that detection systems relying solely on syntactic or stylistic markers may be inadequate for identifying AI-generated content that closely resembles human writing [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredAs AI-generated content becomes more common, institutions rely on detection tools to uphold academic integrity.
“As AI-generated content becomes more common, institutions rely on detection tools to uphold academic integrity [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction links increasing use of AI writing tools to a need to preserve academic integrity, and the Conclusion says detection tools can safeguard the integrity of admissions. But the paper proposes their use in admissions; it does not establish that institutions already rely on them broadly to uphold academic integrity. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe ethical implications of AI detection systems cannot be divorced from broader surveillance, fairness, and justice issues within academia.
“The ethical implications of AI detection systems cannot be divorced from broader surveillance, fairness, and justice issues within academia [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction and Conclusion (pages not provided) connect AI detection in admissions to fair evaluation and academic integrity, and the Related work section says ethical considerations around AI writing tools are not clear-cut. The paper does not connect detection systems to broader surveillance or justice issues within academia, so those parts of the attributed claim are unsupported. (LLM, conf=0.83)
Experimental — accuracy not yet measuredMost institutions cannot afford access to large training datasets, which raises equity and feasibility concerns in implementing fair AI detection systems.
“That is something most institutions cannot afford, which raises equity and feasibility concerns in implementing fair AI detection systems [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Conclusion (page unavailable) says domain-specific detectors require training data that may not always be readily available. It does not say that most institutions cannot afford large datasets or that this creates equity concerns. The Methods section (page unavailable) instead says domain-specific detection is feasible even with limited data resources. (LLM, conf=0.86)
Experimental — accuracy not yet measuredTraditional essay-based assessments, already susceptible to plagiarism, become even more vulnerable to manipulation with AI tools.
“Traditional essay-based assessments, already susceptible to plagiarism, become even more vulnerable to manipulation with AI tools [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
The Introduction (page unavailable) says AI-generated or AI-revised application essays challenge admissions committees’ ability to evaluate applicants’ work. Related work (page unavailable) links AI-text detection and plagiarism detection to concerns about student cheating. The paper focuses on admissions materials, however, and does not establish that traditional essay-based assessments have become more vulnerable than they were to plagiarism. (LLM, conf=0.84)
Experimental — accuracy not yet measuredA central issue is the risk of false accusations when these tools incorrectly flag human-written work as AI-generated.
“A central issue is the risk of false accusations when these tools incorrectly flag human-written work as AI-generated [8,25].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
As generative AI tools like ChatGPT become increasingly capable of producing highly realistic and polished text, traditional essay-based assessments face even greater challenges.
“As generative AI tools like ChatGPT become increasingly capable of producing highly realistic and polished text, traditional essay-based assessments, already prone to issues like plagiarism, face even greater challenges [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
In higher education settings, especially in contexts such as college admissions, distinguishing between genuine student writing and AI-assisted content has become increasingly difficult.
“In higher education settings, especially in contexts such as college admissions, distinguishing between genuine student writing and AI-assisted content has become increasingly difficult [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
A primary concern is the accuracy of detection systems.
“A primary concern is the accuracy of detection systems [8].”
From Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education (2025) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024 · see citing works
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048 · see citing works
ChatGPT User Experience: Implications for Education
Zhai, X. (2023) · doi:10.2139/ssrn.4312418 · Cited by 935 · cited 1× · see citing works
“Task design involves creating assignments to thwart the effectiveness of AI text generation tools 19 while institutional policy relies on guidelines 14 and education 20 to prevent the unethical use of such tools.”
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607 · cited 1× · see citing works
“Task design involves creating assignments to thwart the effectiveness of AI text generation tools 19 while institutional policy relies on guidelines 14 and education 20 to prevent the unethical use of such tools.”
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106 · see citing works
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40 · see citing works
Flagged first, then by citations
Research works printed but not linked: 24
- [2]Westerlund, M. The emergence of deepfake technology: A review. Technol. Innov. Manag. Rev.9 (2019).
- [3]Zellers, R. et al. Defending against neural fake news. Adv. Neural Inf, Process. Syst.32 (2019).
- [4]Ragot, M., Martin, N. & Cojean, S. AI-generated vs. human artworks. a perception bias towards artificial intelligence? In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, 1–10 (2020).
- [5]Pegoraro, A., Kumari, K., Fereidooni, H. & Sadeghi, A.-R. To chatgpt, or not to chatgpt: That is the question! arXiv preprint arXiv:2304.01487 (2023). · arXiv
- [7]Guo, B. et al. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597 (2023). · arXiv
- [10]Berrar, D. Bayes’ theorem and naive bayes classifier. Encycl. Bioinform. Comput. Biol. ABC Bioinform. 403, 412 (2018).
- [12]Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
- [13]Sanh, V., Debut, L., Chaumond, J. & Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv:1910.01108 (2020). · arXiv
- [14]Perkins, M. Academic integrity considerations of AI large language models in the post-pandemic era: Chatgpt and beyond. J. Univ. Teach. Learn. Pract. 20, 07 (2023).
- [15]Dwivedi, Y. K. et al. “so what if chatgpt wrote it?’’ multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manag. 71, 102642 (2023). · DOI · Paper page
- [21]Crothers, E., Japkowicz, N. & Viktor, H. L. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access (2023).
- [23]Nguyen-Son, H.-Q., Tieu, N.-D. T., Nguyen, H. H., Yamagishi, J. & Zen, I. E. Identifying computer-generated text using statistical analysis. In 2017 Asia-Pacific Signal and Info. Processing Association Annual Summit and Conference, 1504–1511 (IEEE, 2017).
- [24]Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D. & Finn, C. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305 (2023). · arXiv
- [25]Solaiman, I. et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203 (2019). · arXiv
- [26]Liu, Y. et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). · arXiv
- [28]Tourille, J., Sow, B. & Popescu, A. Automatic detection of bot-generated tweets. In Proceedings of the 1st International Workshop on Multimedia AI against Disinformation, 44–51 (2022).
- [29]Stiff, H. & Johansson, F. Detecting computer-generated disinformation. Int. J. Data Sci. Anal. 13, 363–383 (2022). · DOI · Paper page
- [32]Von Ahn, L., Blum, M., Hopper, N. J. & Langford, J. Captcha: Using hard AI problems for security. In Advances in Cryptology-EUROCRYPT 2003: International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4–8, 2003 Proceedings 22, 294–311 (Springer, 2003).
- [34]Leskovec, J., Rajaraman, A. & Ullman, J. D. Importance of Words in Documents (page 9) (Cambridge University Press, 2022).
- [35]Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding (2019). arXiv:1810.04805. · arXiv
- [36]Rahman, M. W. U. et al. A bert-based deep learning approach for reputation analysis in social media. In 2022 IEEE/ACS 19th International Conference on Computer Systems and Applications (AICCSA), 1–8 (IEEE, 2022).
- [38]Nayak, A., Timmapathini, H., Ponnalagu, K. & Venkoparao, V. G. Domain adaptation challenges of bert in tokenization and sub-word representations of out-of-vocabulary words. In Proc. 1st Workshop on Insights from Negative Results in NLP, 1–5 (2020).
- [42]Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should i trust you?” explaining the predictions of any classifier. In Proc. of 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144 (2016).
- [43]Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst.30 (2017).
Cited by
Sorted by citations
Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education
Deep, P. D., Edgington, W. D., Ghosh, N., & Rahaman, M. S. (2025) · Information · doi:10.3390/info16100905 · Cited by 26 · see citing works
- Authors
- Zhao, Y., Borelli, A., Martinez, F., Xue, H., & Weiss, G. M.
- Affiliations
- Fordham University (US)
- Corresponding author
- Zhao, Yijun
- Weiss, Gary M.
- ORCID
- Weiss, Gary M.
- Venue
- Scientific Reports
- Published
- 2024-11-02
- Volume
- 14
- Issue
- 1
- Pages
- 26411-26411
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC11531587
- OpenAlex
- W4403993182
- Cited by
- 16
- References
- 6
- Citations by year
- 2026: 9 · 2025: 7
- FWCI
- 2.36
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Scientific Reports (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- AI-generated text detection, ChatGPT, large language models, postgraduate admission, letters of recommendation, text classification, word frequency analysis, GPT-3.5-turbo, admissions equity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
