Augmenting assessment with AI coding of online student discourse: A question of reliability
Open accessAbstract
Currently, many generative Artificial Intelligence (AI) tools are being integrated into the educational technology landscape for instructors. Our paper examines the potential and challenges of using Large Language Models (LLMs) to code student-generated content in online discussions based on intended learning outcomes and how instructors could use this to assess the intended and enacted learning design. If instructors were to rely on LLMs as a means of assessment, the reliability of these models to code the data accurately is crucial. Employing a diverse set of LLMs from the GPT family and prompting techniques on an asynchronous online discussion dataset from a blended-learning bachelor-level course, our research examines the reliability of AI-supported coding in educational research. Findings reveal that while AI-supported coding demonstrates efficiency, achieving substantial, moderate agreement with human coding for specific nuanced and context-dependent codes is challenging. Moreover, the high cost, token limits, and the advanced necessary skills needed to write API scripts might limit the usability of AI-driven coding. Finally, implementation would require specific parameterization techniques based on the class and may not be feasible for widespread implementation. Our study underscores the importance of transparency in AI coding methodologies and the need for a hybrid approach that integrates human judgment to ensure data accuracy and interpretability. In addition, it contributes to the knowledge base about the reliability of LLMs to code real, small datasets using complex codes that are common in the instructor's practice and explores the potential and challenges of using these models for assessment purposes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Manual scoring was employed due to concerns about limited accuracy of LLMs in reliably grading student writing.
“Manual scoring was employed due to concerns regarding the limited accuracy of LLMs, including GPT, in reliably grading student writing (Lee et al., 2024b; Misiejuk et al., 2024).”
From Generative AI and Essay Writing: Impacts of Automated Feedback on Revision Performance and Engagement (2024) · cites this paper as Misiejuk et al., 2024
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · see citing works
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · see citing works
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · see citing works
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 935 · see citing works
Belotto, M. (2018) · The Qualitative Report · doi:10.46743/2160-3715/2018.3492 · Cited by 379 · see citing works
Chicco, D., Warrens, M., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 365 · see citing works
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324 · see citing works
Unveiling security, privacy, and ethical concerns of ChatGPT
Wu, X., Duan, R., & Ni, J. (2023) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2023.10.007 · Cited by 320 · see citing works
Towards artificial intelligence-based assessment systems
Luckin, R. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0028 · Cited by 303 · see citing works
SENS: Network analytics to combine social and cognitive perspectives of collaborative learning
Gašević, D., Joksimovíc, S., Eagan, B. R., & Shaffer, D. W. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.003 · Cited by 227 · see citing works
Augmenting Qualitative Text Analysis with Natural Language Processing: Methodological Study
Guetterman, T. C., Chang, T., DeJonckheere, M. J., Basu, T., Scruggs-Wodkowski, E. A., & Vydiswaran, V. G. V. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9702 · Cited by 181 · see citing works
Using natural language processing technology for qualitative data analysis
Crowston, K., Allen, E. E., & Heckman, R. (2011) · International Journal of Social Research Methodology · doi:10.1080/13645579.2011.625764 · Cited by 169 · see citing works
Chen, N.-C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185515 · Cited by 152 · see citing works
The Handbook of Learning Analytics
Gašević, D., & Merceron, A. (2022) · Solar eBooks · doi:10.18608/hla22 · Cited by 114 · see citing works
Gwet's AC1 is not a substitute for Cohen's kappa – A comparison of basic properties
Vach, W., & Gerke, O. (2023) · MethodsX · doi:10.1016/j.mex.2023.102212 · Cited by 90 · see citing works
Wu, J., Hsiao, Y.-C., & Nian, M.-W. (2018) · Interactive Learning Environments · doi:10.1080/10494820.2018.1515085 · Cited by 88 · see citing works
Kaliisa, R., & Dolonen, J. A. (2022) · Technology Knowledge and Learning · doi:10.1007/s10758-022-09598-7 · Cited by 63 · see citing works
Self-citationFrom nCoder to ChatGPT: From Automated Coding to Refining Human Coding
Zambrano, A. F., Liu, X., Barany, A., Baker, R. S., Kim, J., & Nasiar, N. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47014-1_32 · Cited by 54 · see citing works
Justification for the Use of Cohen’s Kappa Statistic in Experimental Studies of NLP and Text Mining
Kolesnyk, A., & Khairova, N. (2022) · Cybernetics and Systems Analysis · doi:10.1007/s10559-022-00460-3 · Cited by 31 · see citing works
Disagreement on Agreement: Two Alternative Agreement Coefficients
Spratt, K. F. (2007) · Cited by 30 · see citing works
Proceedings of the Third Workshop on Narrative Understanding
(2021) · doi:10.18653/v1/2021.nuse-1 · Cited by 30 · see citing works
Topic-wise classification of MOOC discussions: a visual analytics approach
Atapattu, T., Falkner, K. E., & Tarmazdi, H. (2016) · Educational Data Mining · Cited by 27 · see citing works
Internet of Things and Cyber Physical Systems
Kaushik, K., Dahiya, S., Bhardwaj, A., & Maleh, Y. (2022) · CRC Press eBooks · doi:10.1201/9781003283003 · Cited by 27 · see citing works
Modeling Educational Discourse with Natural Language Processing
Dowell, N., & Kovanović, V. (2022) · Solar eBooks · doi:10.18608/hla22.011 · Cited by 25 · see citing works
Better to be in agreement than in bad company
Silveira, P. S. P., & Siqueira, J. D. O. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01950-0 · Cited by 18 · see citing works
Ostaeyen, S. V., Langhe, L. D., Clercq, O. D., Embo, M. P. C., Schellens, T., & Valcke, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.1056 · Cited by 14 · see citing works
Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums
Yu, J., Alrajhi, L., Harit, A., Sun, Z., Cristea, A. I., & Shi, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80421-3_10 · Cited by 12 · see citing works
Liu, Z., Kong, W., Liu, S., & Kong, X. (2021) · International Conference of Educational Innovation through Technology (EITT) · doi:10.1109/eitt53287.2021.00053 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Rodríguez-Ortiz, M. Á., Santana-Mancilla, P. C., & Anido, L. (2025) · Applied Sciences · doi:10.3390/app15158679 · Cited by 122 · see citing works
RETRACTED: The impact of generative AI on essay revisions and student engagement
Lo, N., Wong, A., & Chan, S. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100249 · Cited by 78 · see citing works
RetractedMzwri, K., & Turcsányi-Szabó, M. (2025) · Education Sciences · doi:10.3390/educsci15020199 · Cited by 70 · see citing works
Chan, S. T. S., Lo, N. P. K., & Wong, A. M. H. (2024) · Contemporary Educational Technology · doi:10.30935/cedtech/15607 · Cited by 56 · see citing works
Kaliisa, R., Misiejuk, K., López‐Pernas, S., & Saqr, M. (2025) · Educational Psychology · doi:10.1080/01443410.2025.2553639 · Cited by 50 · see citing works
Self-citationChan, S., Lo, N., & Wong, A. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2440182 · Cited by 43 · see citing works
RetractedChan, S., Lo, N., & Wong, A. (2024) · rEFLections · doi:10.61508/refl.v31i3.277514 · Cited by 29 · see citing works
Xu, S., Huang, X., Lo, C. K., Chen, G., & Jong, M. S. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100325 · Cited by 30 · see citing works
Lo, N., Chan, S., & Wong, A. (2025) · SAGE Open · doi:10.1177/21582440251352907 · Cited by 29 · see citing works
AI assistance in peer feedback provision: Pedagogically sound, but minimally adopted
Pozdniakov, S., Brazil, J., Banihashem, S. K., Noroozi, O., Gašević, D., Sadiq, S., & Khosravi, H. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105591 · Cited by 11 · see citing works
Generative AI in Learning Design: A Systematic Review
Alkhasawneh, S. N., Λαμπρόπουλος, Γ., & Hernández‐Leo, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98459-4_2 · Cited by 12 · see citing works
Wang, C., & Xiao, J. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2024.2442706 · Cited by 9 · see citing works
AI vs. Human Writing: Developing a Novel Method for Text Authenticity Detection in Education
Department of Computer Science and Engineering, R. I. O. T., Kalmani, V. H., Adamuthe, A. C., Premnath Gondil, A., Prashant Patil, V., Amar Kore, R., & Mahadev Metkari, V. (2025) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2025.03.04 · Cited by 7 · see citing works
Kudritskaya, M. I., Анатольевна, П. Н., Kushnina, L. V., Plekhanova, Y., Matytcina, M. S., & Stepanova, M. (2024) · International Conference on Technology Enhanced Learning in Higher Education (TELE) · doi:10.1109/tele62556.2024.10605668 · Cited by 7 · see citing works
Hybrid Learning, Artificial Intelligence, and Indian Indigenized Values
Makhija, R., Aggarwal, S., & Jain, S. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-9296-6.ch013 · Cited by 6 · see citing works
Kotamjani, S. S., Shirinova, S., Muratova, K., & Sharma, M. (2024) · International Conference on Future Networks and Distributed Systems (ICFNDS) · doi:10.1145/3726122.3726266 · Cited by 5 · see citing works
An Introduction to Large Language Models in Education
Oliveira, E., Song, Y., Saqr, M., & López‐Pernas, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-95365-1_8 · Cited by 5 · see citing works
Exploring GenAI Technologies within Collaborative Learning
Anderson, E., Lin, G. C., Farid, A., Fenech, M., Hanks, B., Klopfer, E. D., Doherty, E. J., Hirshfield, L., Ko, M. M., Foltz, P. W., et al. (2025) · Computer-supported collaborative learning/The Computer-Supported Collaborative Learning Conference · doi:10.22318/cscl2025.991659 · Cited by 6 · see citing works
Educational technology and assessment: a comprehensive bibliometric analysis (2010–2024)
Fiskawarni, T. H., Nurfadilah, N., Saad, R., & Bancong, H. (2025) · Discover Education · doi:10.1007/s44217-025-00872-1 · Cited by 4 · see citing works
Zaus, M. A., Andriani, C., Jasril, I. R., Zaus, A. A., Islami, S., & Yee, S. K. (2025) · Seminars in Medical Writing and Education · doi:10.56294/mw2025820 · Cited by 3 · see citing works
Ioan, A., Rosner, D. E., & Radovici, A. (2025) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs66924.2025.00099 · Cited by 3 · see citing works
Misiejuk, K., López‐Pernas, S., Kaliisa, R., & Saqr, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5145012 · Cited by 3 · see citing works
Self-citationchatgptscrapeR: A Tool for Retrieving Student-AI Interactions
López‐Pernas, S., Misiejuk, K., Jovanović, J., Raspopović, M., Conde, M. Á., & Saqr, M. (2025) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt64023.2025.00044 · Cited by 2 · see citing works
Self-citationCao, L. Y., Etemadi, A., Wheeler, M., & Dede, C. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2568526 · Cited by 1 · see citing works
Using BERT-like Language Models for Automated Discourse Coding: A Primer and Tutorial
López‐Pernas, S., Misiejuk, K., & Saqr, M. (2025) · Springer eBooks · doi:10.1007/978-3-031-95365-1_10 · Cited by 1 · see citing works
Self-citationRughiniş, C., Vulpe, S., & Flaherty, M. G. (2025) · RoEduNet Conference: Networking in Education and Research (RoEduNet) · doi:10.1109/roedunet68395.2025.11208348 · Cited by 1 · see citing works
Kaşarcı, İ. (2026) · International Journal of Current Educational Studies · doi:10.46328/ijces.315 · Cited by 1 · see citing works
Karaköse, T., & Polat, H. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-82915-4_18 · Cited by 1 · see citing works
BAIZHANOV, N. A. (2024) · Pedagogy and Psychology · doi:10.51889/2960-1649.2024.59.2.012
Implementation of System Code Generation Based on Large Language Model Fine-tune with QLoRA
Sya’bani, M. R. S., Richasdy, D., & Kusumo, D. S. (2025) · doi:10.1109/icoict66265.2025.11193032
Xiao, J. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2632766
Lo, C. K., Yeung, L. C., Huang, X., Xu, S., Song, Y., & Chen, G. (2026) · Lecture notes in educational technology · doi:10.1007/978-981-95-8824-4_15
Embedding Cultural Intelligence in AI-Driven Learning Analytics
Jose, M. J., & Mathews, J. R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9898-3.ch009
Boettinger, J., Bürgel, C., & Bartsch, A. (2026) · Quality & Quantity · doi:10.1007/s11135-026-03036-x
A Review of Learning Analytics in GenAI-Assisted Writing
Zhang, T., Lin, W., & Hu, X. (2026) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt68022.2026.00048
Untitled work
Chan, C. K. Y., Lo, N., & Wong, G. (2024)
- Authors
- Misiejuk, K., Kaliisa, R., & Scianna, J.
- Affiliations
- University of Bergen (NO)
- University of Oslo (NO)
- University of Wisconsin–Madison (US)
- Corresponding author
- Misiejuk, Kamila
- Venue
- Computers and Education: Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 6
- Pages
- 100216
- ISSN
- 2666-920X (ISSN-L)
- OpenAlex
- W4393132022
- Cited by
- 33
- References
- 32
- Citations by year
- 2026: 6 · 2025: 21 · 2024: 6
- FWCI
- 12.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- large language models, asynchronous online discussion, coding reliability, interobserver agreement, automated assessment, generative AI in education, blended learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
