CoChat
Sign Up
PreprintCited by 39EFL/ESL Teaching and LearningArtificial Intelligence in EducationEnglish Language Learning and Teaching

The DOI matches its DataCite record.

Exploring AI-Generated Text in Student Writing: How Does AI Help?

Open access
Woo, David James, Susanto, Hengky, Yeung, Chi Ho, Guo, Kai, Fung, April Ka Yeng·2023·arXiv
DOI 10.48550/arxiv.2304.02478arXiv 2304.02478
Open this paper in CoChat — free accountOpen PDF

Abstract

English as foreign language_EFL_students' use of text generated from artificial intelligence_AI_natural language generation_NLG_tools may improve their writing quality. However, it remains unclear to what extent AI-generated text in these students' writing might lead to higher-quality writing. We explored 23 Hong Kong secondary school students' attempts to write stories comprising their own words and AI-generated text. Human experts scored the stories for dimensions of content, language and organization. We analyzed the basic organization and structure and syntactic complexity of the stories' AI-generated text and performed multiple linear regression and cluster analyses. The results show the number of human words and the number of AI-generated words contribute significantly to scores. Besides, students can be grouped into competent and less competent writers who use more AI-generated text or less AI-generated text compared to their peers. Comparisons of clusters reveal some benefit of AI-generated text in improving the quality of both high-scoring students' and low-scoring students' writing. The findings can inform pedagogical strategies to use AI-generated text for EFL students' writing and to address digital divides. This study contributes designs of NLG tools and writing activities to implement AI-generated text in schools.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.02478

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Woo et al. (2024) found high creative output.

    “A high creative output and the self-guided integration of AI-generated text showed strong gains in both agency and creative expression.”

    From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [6]

    In “Scoring of Stories” (page unavailable), creativity is one component of the content score; the Figure 7 discussion reports relatively high content scores for some students who incorporated substantial AI-generated text. That supports high-rated creative writing in some cases, but the paper does not establish strong gains in creative expression or agency from self-guided AI integration. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Woo et al. (2024) found self-guided integration of AI-generated text showed strong gains in both agency and creative expression.

    “A high creative output and the self-guided integration of AI-generated text showed strong gains in both agency and creative expression.”

    From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [6]

    In “This Study” (page unavailable), the authors describe students choosing whether to integrate AI-generated text, which supports the self-guided aspect. Figure 7 reports high content scores for some students who used substantial AI-generated text, but also low scores for others. The paper does not report strong gains in agency and creative expression; agency is a feature of the study design, not a measured improvement. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Woo et al. demonstrated how generative AI tools helped students overcome creative barriers in story writing across various genres.

    “Similarly, Woo et al. [6] demonstrated how generative AI tools helped students overcome creative barriers in story writing across various genres.”

    From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [6]

    The Abstract (page unavailable) reports that 23 students used AI tools to write stories and that AI-generated words contributed to writing scores. It does not establish that the tools helped students overcome creative barriers or that the stories spanned various genres. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Higher-performing students demonstrated greater agency by selectively incorporating AI output to enhance content and organization, rather than relying on it passively.

    “Higher-performing students, in particular, demonstrated greater agency by selectively incorporating AI output to enhance content and organization, rather than relying on it passively.”

    From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [6]

  • Woo et al.'s study revealed that students made independent and strategic decisions about how to use AI.

    “By analyzing student writing products, which distinguished between human-and AI-written text, and applying rubric-based evaluations alongside cluster analysis, the study revealed that students made independent and strategic decisions about how to use AI.”

    From AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review (2025) · cites this paper as [6]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Aiken, L. S., West, S. G., & Pitts, S. C. (2003). Multiple linear regression. Handbook of psychology, 481-507. https://doi.org/10.1002/0471264385. · DOI
  2. wei0219 · DOI
  3. Ali, S., DiPaola, D., Lee, I., Hong, J., & Breazeal, C. (2021). Exploring Generative Models with Middle School Students. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1-13. https://doi.org/10.1145/3411764.3445226 · DOI · Paper page
  4. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. https://doi.org/10.1145/3442188.3445922 · DOI · Paper page
  5. Brown, B. L., & Hendrix, S. B. (2005). Partial correlation coefficients. Encyclopedia of statistics in behavioral science. https://doi.org/10.1002/0470013192. · DOI
  6. bsa469 · DOI
  7. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners (arXiv:2005.14165). arXiv. https://doi.org/10.48550/arXiv.2005.14165 · DOI · Paper page
  8. Butterfuss, R., Roscoe, R. D., Allen, L. K., McCarthy, K. S., & McNamara, D. S. (2022). Strategy Uptake in Writing Pal: Adaptive Feedback and Instruction. Journal of Educational Computing Research, 60(3), 696-721. https://doi.org/10.1177/07356331211045304 · DOI · Paper page
  9. Calderwood, A., Qiu, V., Gero, K. I., & Chilton, L. B. (2020). How Novelists Use Generative Language Models: An Exploratory User Study. IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces. ACM IUI 2020, Cagliari, Italy.
  10. Chiu, T. K. F. (2021). A Holistic Approach to the Design of Artificial Intelligence (AI) Education for K-12 Schools. TechTrends, 65(5), 796-807. https://doi.org/10.1007/s11528-021-00637-1 · DOI · Paper page
  11. Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018). Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. 23rd International Conference on Intelligent User Interfaces, 329-340. https://doi.org/10.1145/3172944.3172983 · DOI · Paper page
  12. Crossley, S. A., Muldner, K., & McNamara, D. S. (2016). Idea Generation in Student Writing: Computational Assessments and Links to Successful Writing. Written Communication, 33(3), 328-354. https://doi.org/10.1177/0741088316650178 · DOI
  13. Crossley, S. A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), Article 3. https://doi.org/10.17239/jowr- 2020.11.03.01 · DOI · Paper page
  14. DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2010). Developing and Using a Codebook for the Analysis of Interview Data: An Example from a Professional Development Research Project. Field Methods, 23(2), 136-155. https://doi.org/10.1177/1525822X10388468 · DOI · Paper page
  15. Dekking, F. M., Kraaikamp, C., Lopuhaä, H. P., & Meester, L. E. (2005). A Modern Introduction to Probability and Statistics: Understanding why and how (Vol. 488). London: Springer. https://link.springer.com/book/10.1007/1-84628-168-7 · DOI
  16. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1-22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x · DOI · Paper page
  17. Dizon, G. (2020). Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology, 24(1), 16-26. https://doi.org/10125/44705
  18. Dizon, G., & Gayed, J. (2021). Examining the impact of Grammarly on the quality of mobile L2 writing. The JALT CALL Journal, 17(2), 74-92. https://doi.org/10.29140/jaltcall.v17n2.336 · DOI · Paper page
  19. Flower, L., & Hayes, J. R. (1981). A Cognitive Process Theory of Writing. College Composition and Communication, 32(4), 365-387. https://doi.org/10.2307/356600 · DOI · Paper page
  20. Fukunaga, K., & Hostetler, L. (1975). The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory, 21(1), 32- 40. https://doi.org/10.1109/TIT.1975.1055330 · DOI
  21. Gayed, J. M., Carlon, M. K. J., Oriola, A. M., & Cross, J. S. (2022). Exploring an AI-based writing Assistant's impact on English language learners. Computers and Education: Artificial Intelligence, 3, 100055. https://doi.org/10.1016/j.caeai.2022.100055 · DOI · Paper page
  22. Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning & Technology, 26(2), 5-24. http://doi.org/10125/73474
  23. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration (arXiv:1904.09751). arXiv. http://arxiv.org/abs/1904.09751 · arXiv
  24. Hyland, K. (2003a). Second Language Writing. Cambridge University Press.
  25. Hyland, K. (2003b). Genre-based pedagogies: A social response to process. Journal of Second Language Writing, 12(1), 17-29. https://doi.org/10.1016/S1060-3743(02)00124-8 · DOI
  26. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Dai, W., Madotto, A., & Fung, P. (2022). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 3571730. https://doi.org/10.1145/3571730 · DOI · Paper page
  27. Kangasharju, A., Ilomäki, L., Lakkala, M., & Toom, A. (2022). Lower secondary students' poetry writing with the AI-based Poetry Machine. Computers and Education: Artificial Intelligence, 3, 100048. https://doi.org/10.1016/j.caeai.2022.100048 · DOI
  28. Lin, M. P.-C., & Chang, D. (2020). Enhancing Post-secondary Writers' Writing Skills with a Chatbot: A Mixed-Method Classroom Study. Journal of Educational Technology & Society, 23(1), 78-92. https://www.jstor.org/stable/26915408
  29. Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474-496. https://doi.org/10.1075/ijcl.15.4.02lu MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, 5.1, 281-298. https://projecteuclid.org/ebooks/berkeley-symposium- on-mathematical-statistics-and-probability/Proceedings-of-the-Fifth-Berkeley- Symposium-on-Mathematical-Statistics-and/chapter/Some-methods-for-classification- and-analysis-of-multivariate-observations/bsmsp/1200512992 · DOI
  30. Prior, P. A. (2006). A Sociocultural Theory of Writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), The Handbook of Writing Research (pp. 54-66). Guilford Press.
  31. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. Undefined. https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask- Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe
  32. Reynolds, L., & McDonell, K. (2021). Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (arXiv:2102.07350). arXiv. https://doi.org/10.48550/arXiv.2102.07350 · DOI
  33. Tan, X. (2023). Stories behind the scenes: L2 students' cognitive processes of multimodal composing and traditional writing. Journal of Second Language Writing, 59, 100958. https://doi.org/10.1016/j.jslw.2022.100958 · DOI
  34. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (pp. xi, 159). Harvard U Press.
  35. Williams, R., Ali, S., Devasia, N., DiPaola, D., Hong, J., Kaputsos, S. P., Jordan, B., & Breazeal, C. (2022). AI + Ethics Curricula for Middle School Youth: Lessons Learned from Three Project-Based Curricula. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00298-y · DOI
  36. Yang, D., Zhou, Y., Zhang, Z., & Li, T. J.-J. (2022). AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. Joint Proceedings of the ACM IUI Workshops 2022, 10. Note.

Cited by

Sorted by citations

Metadata

Authors
Woo, D. J., Susanto, H., Yeung, C. H., Guo, K., & Fung, A. K. Y.
Affiliations
  • Education University of Hong Kong (HK)
  • Department of Science and Environmental Studies, The Education University of Hong Kong, Hong Kong
  • University of Hong Kong (HK)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
28
Issue
2
Pages
183-209
ISSN
1094-3501 (ISSN-L)
DOI
10.48550/arxiv.2304.02478
arXiv
2304.02478
OpenAlex
W4413183854
Cited by
39
References
0
Citations by year
2026: 16 · 2025: 19 · 2024: 4
FWCI
61.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Language learning & technology (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • ScholarSpace (University of Hawaii at Manoa) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • Education University of Hong Kong (FLASS/DRF 04418, FLASS/DRF 04624, FLASS/ROP 04396)
  • Faculty of Liberal Arts and Social Sciences, Education University of Hong Kong (FLASS/ROP 04396, FLASS/DRF 04418, FLASS/DRF 04624)
Topics
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
  • Social Sciences › Education › Artificial Intelligence in Education
  • Social Sciences › Education › English Language Learning and Teaching
Keywords
AI-generated text, AI writing tools, EFL writing, student writing quality, syntactic complexity, writing organization, writing skills, high school students, writing instruction strategies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review