Evidence-centered Assessment for Writing with Generative AI
Open accessAbstract
We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we used epistemic network analysis to make inferences from the data about the claims. Our findings revealed significant differences in the writing processes of different groups of CoAuthor users, suggesting that our method is a plausible approach to assessing human-AI collaborative writing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cheng et al. (2024) revealed significant differences in writing processes across groups defined by authorship (User vs. GAI), writing type (Creative vs. Argumentative), and GPT temperature settings (High vs. Low).
“Cheng et al. (2024) proposed a learning analytics-based methodology for evaluating GAI-assisted writing, revealing significant differences in writing processes across groups defined by authorship (User vs. GAI), writing type (Creative vs. Argumentative) and GPT temperature settings (High vs. Low).”
From Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing (2026) · cites this paper as Cheng et al., 2024
The Discussion and Conclusions reports an assessment method and statistically significant writing-process differences for user versus GAI ownership and creative versus argumentative prompts. The Regression Results (section 4.2; page not provided) says temperature was not statistically significant on either dimension, so the claimed significant High-versus-Low temperature difference is not supported. (LLM, conf=0.94)
Experimental — accuracy not yet measuredENA has demonstrated its effectiveness and validity as a method for assessing GAI-assisted writing in previous research.
“ENA has demonstrated its effectiveness and validity as a method for assessing GAIassisted writing in previous research (Cheng et al., 2024).”
From Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing (2026) · cites this paper as Cheng et al., 2024
The Discussion and Conclusions report statistically significant differences between writing conditions using an ENA-based assessment, supporting its usefulness as a proof of concept. But the Research Questions section says the dataset lacked writing-quality scores needed to test whether the method distinguishes high- from low-quality processes, and the Discussion calls the method plausible rather than validated. The claim overstates the evidence for validity. (LLM, conf=0.87)
Experimental — accuracy not yet measuredIn the Creative context, support for reviewing features a mix of quantitative and qualitative feedback, with a focus on the AI evaluating text and providing suggestions rather than revising it directly.
“Support for reviewing features a mix of quantitative and qualitative feedback, with a focus on the AI evaluating text and providing suggestions rather than revising it directly [20,126].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [20]
The ECD Task Model (§2.2; page unavailable) says writers can request AI suggestions and choose whether to accept or modify them. But the Discussion and Conclusions (§5; page unavailable) specifies that these suggestions only continue the current text. The paper does not describe AI reviewing or evaluating text, or providing a mix of quantitative and qualitative feedback. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPrevious studies on GAI-assisted writing often categorize writers based on overall AI usage throughout the entire writing session, overlooking temporal dynamics.
“However, these existing studies often have two primary shortcomings: (i) They generally categorized writers based on their overall AI usage throughout the entire writing session, overlooking the temporal dynamics of these behaviours as the progression of a writing session (studies by Cheng et al., 2024;Shibani et al., 2023;Wambsganss et al., 2023;Yang et al., 2025);”
From Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing (2026) · cites this paper as Cheng et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373 · see citing works
<b>lmerTest</b> Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346 · see citing works
Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education
Garrison, D. R., Anderson, T., & Archer, W. (1999) · The Internet and Higher Education · doi:10.1016/s1096-7516(00)00016-6 · Cited by 6,619 · see citing works
Clark, A., & Chalmers, D. (1998) · Analysis · doi:10.1093/analys/58.1.7 · Cited by 5,457 · cited 1× · see citing works
“A key feature of assessments is that they assign credit to both pragmatic actions—actions that advance toward a goal state—and epistemic actions—actions that reduce cognitive effort [3].”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“…to maintain a reality status; and (4) authors responding to argumentative writing prompts would have writing processes more characterised by knowledge transformation, integration, and resolution as they may have needed to align GAI outputs (which are known to include falsehoods [34]) with reality.”
BackgroundAn Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
The Psychology of Written Composition
Cumming, A. H., Bereiter, C., & Scardamalia, M. (1989) · Modern Language Journal · doi:10.2307/327270 · Cited by 2,806 · see citing works
Semi-Markov Processes and Reliability
Limnios, N., & Oprişan, G. (2001) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-0161-8 · Cited by 555 · see citing works
Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes
Leijten, M., & Van Waes, L. (2013) · Written Communication · doi:10.1177/0741088313491692 · Cited by 513 · see citing works
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409 · cited 2× · see citing works
“Much of the worry surrounding assessment and GAI is centered on the only evidence of learning being a product with unknown providence [13].”
BackgroundMethodSelf-citationEpistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics
University of Wisconsin–Madison, U., Shaffer, D. W., Ruis, A. R., & University of Wisconsin–Madison, U. (2017) · Handbook of Learning Analytics · doi:10.18608/hla17.015 · Cited by 255 · cited 1× · see citing works
“ENA is a widely used learning analytic technique for modeling actions via network models [31].”
MethodAutomated Essay Scoring: A Survey of the State of the Art
Ke, Z., & Ng, V. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/879 · Cited by 247 · see citing works
On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation
Wang, Y., Wang, C., Li, R., & Lin, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.249 · Cited by 118 · see citing works
Zuur, A. F., Ieno, E. N., Walker, N. J., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for biology and health · doi:10.1007/978-0-387-87458-6_1 · Cited by 85 · see citing works
Analysis of collaborative writing processes using revision maps and probabilistic topic models
Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A. (2013) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2460296.2460307 · Cited by 69 · see citing works
Fan, Y., Raković, M., van der Graaf, J., Lim, L., Singh, S., Moore, J. D., Molenaar, I., Bannert, M., & Gašević, D. (2023) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12801 · Cited by 65 · see citing works
Self-citationLessons learned from implementing remotely invigilated online exams
Cramp, J., Medlin, J., Lake, P., & Sharp, C. (2019) · Journal of University Teaching and Learning Practice · doi:10.53761/1.16.1.10 · Cited by 51 · see citing works
Raković, M., Iqbal, S., Li, T., Fan, Y., Singh, S., Surendrannair, S., Kilgour, J., van der Graaf, J., Lim, L., Molenaar, I., et al. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12769 · Cited by 36 · see citing works
Self-citationHu, Y., Mello, R. F., & Gašević, D. (2021) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2021.100037 · Cited by 32 · see citing works
Self-citationArtificial Intelligence in Education
Kamer, S. T. (2024) · CRC Press eBooks · doi:10.1201/9781003483571-15 · Cited by 24 · see citing works
Advances in Quantitative Ethnography
Kim, Y. J., & Swiecki, Z. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-76332-8 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- James V. Wertsch. Mind As Action. Oxford University Press, Incorporated, New York, UNITED STATES, 1998.
- Antonette Shibani, Ming Liu, Christian Rapp, and Simon Knight. Advances in writing analytics: Mapping the state of the field. In Companion Proceedings of the 9th International Conference on Learning Analytics & Knowledge, LAK '19. Association for Computing Machinery, 03 2019.
- Sehrish Iqbal, Mladen Rakovic, Guanliang Chen, Tongguang Li, Rafael Ferreira Mello, Yizhou Fan, Giuseppe Fiorentino, Naif Radi Aljohani, and Dragan Gasevic. Towards automated analysis of rhetorical categories in students essay writings using bloom's taxonomy. In LAK23: 13th International Learning Analytics and Knowledge Conference, LAK2023, page 418-429. Association for Computing Machinery, 2023.
- Mladen Raković. Automatic Identification of Knowledge Transforming Content in Argument Essays Developed from Multiple Sources. Phd thesis, Simon Fraser University, British Columbia, CA, sep 2019.
- Sehrish Iqbal, Zachari Swiecki, Srecko Joksimovic, Rafael Ferreira Mello, Naif Aljohani, Saeed Ul Hassan, and Dragan Gasevic. Uncovering associations between cognitive presence and speech acts: A network-based approach. In LAK22: 12th International Learning Analytics and Knowledge Conference, LAK22, page 315-325. Association for Computing Machinery, 2022.
- Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. CoRR, abs/2201.06796, 2022. · arXiv
- Andrew Gibson and Antonette Shibani. Natural language processing -writing analytics. In The Handbook of Learning Analytics, pages 96-104. SoLAR, 2 edition, 2022.
- H. Guo, M. Zhang, P. Deane, and R. E. Bennett. Writing process differences in subgroups reflected in keystroke logs. Journal of Educational and Behavioral Statistics, 44(5):571-596, 2019.
- Mariëlle Leijten and Luuk Van Waes. Keystroke logging in writing research. Written Communication, 30(3):358-392, 2013.
- Antonette Shibani, Ratnavel Rajalakshmi, Faerie Mattins, Srivarshan Selvaraj, and Simon Knight. Visual representation of co-authorship with GPT-3: Studying human-machine interaction for effective writing. In Proceedings of the 16th International Conference on Educational Data Mining, pages 183-193. International Educational Data Mining Society, July 2023.
- Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. CoRR, abs/1908.10084, 2019. · arXiv
- Dale Bowman, Zachari Swiecki, Zhiqiang Cai, Yeyu Wang, Brendan Eagan, Jeff Linderoth, and David Williamson Shaffer. The mathematical foundations of epistemic network analysis. In Advances in Quantitative Ethnography, Communications in Computer and Information Science, pages 91-105. Springer, 2021.
Cited by
Sorted by citations
Rodríguez-Ortiz, M. Á., Santana-Mancilla, P. C., & Anido, L. (2025) · Applied Sciences · doi:10.3390/app15158679 · Cited by 122 · see citing works
Evaluating the quality of AI feedback: A comparative study of AI and human essay grading
Almegren, A., Mahdi, H. S., Hazaea, A. N., Ali, J. K. M., & Almegren, R. M. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2437122 · Cited by 50 · see citing works
Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37 · see citing works
Supporting learner agency in collaborative writing with generative <scp>AI</scp>
Kim, S., So, H., & Park, K. (2026) · British Journal of Educational Technology · doi:10.1111/bjet.70015 · Cited by 30 · see citing works
Modifying AI, Enhancing Essays: How Active Engagement with Generative AI Boosts Writing Quality
Yang, K., Raković, M., Liang, Z., Yan, L., Zeng, Z., Fan, Y., Gašević, D., & Chen, G. (2025) · Proceedings of the 15th International Learning Analytics and Knowledge Conference · doi:10.1145/3706468.3706544 · Cited by 26 · see citing works
Self-citationJoo, S. H. (2024) · Studies in Applied Linguistics and TESOL · doi:10.52214/salt.v24i1.12865 · Cited by 28 · see citing works
Asking generative artificial intelligence the right questions improves writing performance
Cheng, Y., Fan, Y., Li, X., Chen, G., Gašević, D., & Swiecki, Z. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100374 · Cited by 21 · see citing works
Self-citationInnovations in Online Learning Analytics: A Review of Recent Research and Emerging Trends
Wang, Y., Lai, Y., & Huang, X. (2024) · IEEE Access · doi:10.1109/access.2024.3493621 · Cited by 24 · see citing works
Chen, A., Lian, J., Kuang, X., & Jia, J. (2025) · The Internet and Higher Education · doi:10.1016/j.iheduc.2025.101054 · Cited by 18 · see citing works
Jung, Y., & Jin, S.-H. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2556808 · Cited by 16 · see citing works
Self-regulated Learning Processes in Secondary Education: A Network Analysis of Trace-based Measures
Cheng, Y., Guan, R., Li, T., Raković, M., Li, X., Fan, Y., Jin, F. J.-Y., Tsai, Y., Gašević, D., & Swiecki, Z. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706502 · Cited by 12 · see citing works
Self-citationThe lack of generalisability in learning analytics research: why, how does it matter, and where to?
Khalil, M., & Prinsloo, P. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706489 · Cited by 5 · see citing works
Ink and algorithm: Exploring temporal dynamics in generative <scp>AI</scp> ‐assisted writing
Yang, K., Cheng, Y., Zhao, L., Raković, M., Swiecki, Z., Gašević, D., & Chen, G. (2026) · British Journal of Educational Technology · doi:10.1111/bjet.70031 · Cited by 4 · see citing works
Self-citationEbrahimzadeh, M., Shibani, A., & Shum, S. B. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100609 · Cited by 5 · see citing works
Haddadian, G., Kim, M. K., & Haddadian, N. (2025) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2025.2598266 · Cited by 3 · see citing works
Olivier, C., & Weilbach, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72234-9_7 · Cited by 4 · see citing works
Learners’ Awareness of AI as Originator of an Academic Text During Revision: An Exploratory Study
Radtke, A., Meyer, J., & Rummel, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98417-4_1 · Cited by 1 · see citing works
Sha, L., Liang, Z., Zhao, L., Li, Y., Fernandez-Nieto, G. M., Tsai, Y., Swiecki, Z., Chen, G., & Gašević, D. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2643354 · Cited by 1 · see citing works
Self-citationIncorporating Artificial Intelligence as a Tool in Teaching Writing Skills
Alhiti, S. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-0771-8.ch003 · Cited by 2 · see citing works
Oliveira, M., Zednik, C., Bombaerts, G. J. T., Sadowski, B. M., & Conijn, R. (2025) · doi:10.31234/osf.io/cne9j_v1
Oliveira, M., Zednik, C., Bombaerts, G. J. T., Sadowski, B. M., & Conijn, R. (2025) · doi:10.31234/osf.io/cne9j_v2
SchrijfBlik: Safeguarding the Validity of Writing Assessment in the Age of AI
Poell, T., Maas, L., Balk, M., & van Haastrecht, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99261-2_6 · Cited by 1 · see citing works
More Than Words: Evidencing Qualitative Findings Through Multimodal Narratives
Cheng, Y., & Swiecki, Z. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-12229-2_16
Self-citationVoss, E., Joo, S. H., & Eskin, D. (2026) · Language Assessment Quarterly · doi:10.1080/15434303.2025.2606233 · Cited by 1 · see citing works
Curfman, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6612798
Koller, T., & Keller, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31421-5_6
Abhishek, N. A., Nayak, K. M., Divyashree, M. S., & Jain, U. (2026) · Frontiers of digital education. · doi:10.1007/s44366-026-0102-2
Lupowitz, J. L. (2026) · Tiempo devorado · doi:10.5565/rev/tdevorado.279
How does conversation unfold? analyzing learner–LLM dialogues through multiple analytical lenses
Hidalgo, L., Pérez‐Sanagustín, M., do Nascimento, A. C. A., & Muñoz-Gama, J. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2739434
- Authors
- Cheng, Y., Lyons, K., Chen, G., Gašević, D., & Swiecki, Z.
- Affiliations
- Monash University (AU)
- The University of Melbourne (AU)
- Venue
- Proceedings of the 14th Learning Analytics and Knowledge Conference
- Publisher
- ACM
- Published
- 2024-03-18
- Pages
- 178-188
- OpenAlex
- W4392445354
- Cited by
- 29
- References
- 23
- Citations by year
- 2026: 8 · 2025: 17 · 2024: 4
- FWCI
- 12.72
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Learning Analytics and Knowledge (LAK) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- HAL (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Discovery Projects (220101209, 240100069)
- CELLA 2 CERES (Jacobs Foundation)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- generative AI, learning analytics, epistemic network analysis, cognitive presence, knowledge transformation, writing process
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
