Using Learner Trace Data to Understand Metacognitive Processes in Writing from Multiple Sources
Abstract
Writing from multiple sources is a commonly administered learning task across educational levels and disciplines. In this task, learners are instructed to comprehend information from source documents and integrate it into a coherent written composition to fulfil the assignment requirements. Even though educationally potent, multi-source writing tasks are considered challenging to many learners, in particular because many learners underuse monitoring and control, critical metacognitive processes for productive engagement in multi-source writing. To understand these processes, we conducted a laboratory study involving 44 university students. They engaged in multi-source writing task hosted in digital learning environment. Adding to previous research, we unobtrusively measured metacognitive processes using learners’ trace data collected via multiple data channels and in both writing and reading space of the multi-source writing task. We further investigated how these processes affect the quality of a written product, i.e., essay score. In the analysis, we utilised both automatically and human-generated essay score. The rating performance of the essay scoring algorithm was comparable to that of human raters. Our results largely support the theoretical assumptions that engagement in metacognitive monitoring and control benefits the quality of written product. Moreover, our results can inform the development of analytics-based tools that support student writing by making use of trace data and automated essay scoring.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Coding was performed by experts using theory-driven guidelines and a tracing protocol adapted from validated methods for measuring SRL via trace data.
“Coding was performed by experts using theory-driven guidelines and a tracing protocol adapted from validated methods for measuring SRL via trace data (Fan et al., 2022; Rakovic et al., 2022; Li et al., 2023; van der Graaf et al., 2023a; Lim et al., 2023).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rakovic et al., 2022
Cites
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665 · see citing works
Handbook of Metacognition in Education
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (2009) · doi:10.4324/9780203876428 · Cited by 1,509 · see citing works
An Improved Non-monotonic Transition System for Dependency Parsing
Honnibal, M., & Johnson, M. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1162 · Cited by 618 · see citing works
Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes
Leijten, M., & Van Waes, L. (2013) · Written Communication · doi:10.1177/0741088313491692 · Cited by 513 · see citing works
International Journal of Applied Science and Technology
(2021) · International Journal of Applied Science and Technology · doi:10.30845/ijast · Cited by 495 · see citing works
RESOLV: Readers' Representation of Reading Contexts and Tasks
Rouet, J., Britt, M. A., & Durik, A. M. (2017) · Educational Psychologist · doi:10.1080/00461520.2017.1329015 · Cited by 281 · see citing works
Training Advanced Writing Skills: The Case for Deliberate Practice
Kellogg, R. T., & Whiteford, A. P. (2009) · Educational Psychologist · doi:10.1080/00461520903213600 · Cited by 262 · see citing works
Students' writing from sources for academic purposes: A synthesis of recent research
Cumming, A. H., Lai, C., & Cho, H. (2016) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2016.06.002 · Cited by 245 · see citing works
Shermis, M. D. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.04.001 · Cited by 227 · see citing works
Trends in Research on Writing as a Learning Activity
Klein, P. D., & Boscolo, P. (2016) · Journal of Writing Research · doi:10.17239/jowr-2016.07.03.01 · Cited by 226 · see citing works
Asymptotic Robust Inferences in the Analysis of Mean and Covariance Structures
Satorra, A. (1992) · Sociological Methodology · doi:10.2307/270998 · Cited by 155 · see citing works
AcaWriter: A learning analytics tool for formative feedback on academic writing
Knight, S., Shibani, A., Abel, S., Gibson, A., & Ryan, P. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.12.01.06 · Cited by 151 · see citing works
Handbook of Multiple Source Use
Coiro, J., Sparks, J. R., & Kulikowich, J. M. (2018) · Routledge eBooks · doi:10.4324/9781315627496 · Cited by 148 · see citing works
Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country
Bridgeman, B., Trapani, C. S., & Attali, Y. (2012) · Applied Measurement in Education · doi:10.1080/08957347.2012.635502 · Cited by 144 · see citing works
Barkaoui, K. (2016) · Modern Language Journal · doi:10.1111/modl.12316 · Cited by 113 · see citing works
Flower, L. S., Stein, V., Ackerman, J., Kantz, M., McCormick, K., & Peck, W. (1990) · doi:10.1093/oso/9780195061901.001.0001 · Cited by 105 · see citing works
The assessment and instruction of self-regulation in computer-based environments: a discussion
Veenman, M. V. J. (2007) · Metacognition and Learning · doi:10.1007/s11409-007-9017-6 · Cited by 99 · see citing works
Vandermeulen, N., Leijten, M., & Waes, L. V. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.12.01.05 · Cited by 69 · see citing works
Improving university argumentative writing through an online training
Luna, M., Villalón, R., Mateos, M., & Martín, E. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.12.01.08 · Cited by 61 · see citing works
Sung, Y., Liao, C.-N., Chang, T.-H., Chen, C., & Chang, K.-E. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.12.003 · Cited by 60 · see citing works
Karlen, Y. (2017) · Journal of Writing Research · doi:10.17239/jowr-2017.09.01.03 · Cited by 54 · see citing works
Facilitating metacognitive processes of academic genre-based writing using an online writing system
Yeh, H. (2014) · Computer Assisted Language Learning · doi:10.1080/09588221.2014.881384 · Cited by 51 · see citing works
The academic writing of community college remedial students: Text and learner variables
Perin, D., Keselman, A., & Monopoli, M. (2003) · Higher Education · doi:10.1023/a:1021237532056 · Cited by 51 · see citing works
Wiley, J., Hastings, P., Blaum, D., Jaeger, A. J., Hughes, S., Wallace, P., Griffin, T. D., & Britt, M. A. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0138-z · Cited by 49 · see citing works
Zhang, H., Magooda, A., Litman, D., Correnti, R., Wang, E., Matsmura, L. C., Howe, E., & Quintana, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019619 · Cited by 46 · see citing works
Challenges in Argumentation and Paraphrasing Among Beginning Students in Educational Sciences
Hyytinen, H., Löfstrôm, E., & Lindblom‐Ylänne, S. (2016) · Scandinavian Journal of Educational Research · doi:10.1080/00313831.2016.1147072 · Cited by 38 · see citing works
Automated Summarization Evaluation (ASE) Using Natural Language Processing Tools
Crossley, S. A., Kim, M., Allen, L. K., & McNamara, D. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23204-7_8 · Cited by 36 · see citing works
Assessing the use of multiple sources in student essays
Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. A. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0214-0 · Cited by 27 · see citing works
Zhang, H., & Litman, D. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.759 · Cited by 16 · see citing works
Machine Learning for Holistic Evaluation of Scientific Essays
Hughes, S., Hastings, P., Britt, M. A., Wallace, P. S., & Blaum, D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_17 · Cited by 13 · see citing works
SR-WMS: A Typology of Self-Regulation in Writing from Multiple Sources
Raković, M., & Winne, P. H. (2022) · Advances in analytics for learning and teaching · doi:10.1007/978-3-031-06333-6_7 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
How educational chatbots support self-regulated learning? A systematic review of the literature
Guan, R., Raković, M., Chen, G., & Gašević, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12881-y · Cited by 162 · see citing works
Fan, Y., Raković, M., van der Graaf, J., Lim, L., Singh, S., Moore, J. D., Molenaar, I., Bannert, M., & Gašević, D. (2023) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12801 · Cited by 65 · see citing works
Self-citationAnalytics of self-regulated learning scaffolding: effects on learning processes
Li, T., Fan, Y., Tan, Y., Wang, Y., Singh, S., Li, X., Raković, M., van der Graaf, J., Lim, L., Yang, B., et al. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1206696 · Cited by 44 · see citing works
Self-citationRaković, M., Iqbal, S., Li, T., Fan, Y., Singh, S., Surendrannair, S., Kilgour, J., van der Graaf, J., Lim, L., Molenaar, I., et al. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12769 · Cited by 36 · see citing works
Self-citationQian, K., Liu, S., Li, T., Raković, M., Li, X., Guan, R., Molenaar, I., Nawaz, S., Swiecki, Z., Yan, L., et al. (2026) · Computers & Education · doi:10.1016/j.compedu.2025.105448 · Cited by 39 · see citing works
“Coding was performed by experts using theory-driven guidelines and a tracing protocol adapted from validated methods for measuring SRL via trace data (Fan et al., 2022; Rakovic et al., 2022; Li et al., 2023; van der Graaf et al., 2023a; Lim et al., 2023).”
Self-citationAsking generative artificial intelligence the right questions improves writing performance
Cheng, Y., Fan, Y., Li, X., Chen, G., Gašević, D., & Swiecki, Z. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100374 · Cited by 21 · see citing works
Self-citationSelf-regulated Learning in the Digitally Enhanced Science Classroom: Toward an Early Warning System
Kubsch, M., Strauß, S., Grimm, A., Gombert, S., Drachsler, H., Neumann, K., & Rummel, N. (2025) · Educational Psychology Review · doi:10.1007/s10648-025-10011-9 · Cited by 23 · see citing works
Nguyen, A., Ilesanmi, F., Dang, B., Vuorenmaa, E., & Järvelä, S. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240198 · Cited by 24 · see citing works
Aksela, O., Lämsä, J., & Järvelä, S. (2024) · Technology Knowledge and Learning · doi:10.1007/s10758-024-09789-4 · Cited by 30 · see citing works
Raković, M., Li, Y., Foumani, N. M., Salehi, M., Kuhlmann, L., Mackellar, G., Martínez‐Maldonado, R., Haffari, G., Swiecki, Z., Li, X., et al. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636934 · Cited by 13 · see citing works
Fernandez, J., & Guilbert, J. (2024) · Metacognition and Learning · doi:10.1007/s11409-024-09398-7 · Cited by 21 · see citing works
Iqbal, S., Raković, M., Chen, G., Li, T., Bajaj, J., Mello, R. F., Fan, Y., Aljohani, N. R., & Gašević, D. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636927 · Cited by 10 · see citing works
Self-citationIqbal, S., Raković, M., Chen, G., Li, T., Mello, R. F., Fan, Y., Fiorentino, G., Aljohani, N. R., & Gašević, D. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576112 · Cited by 10 · see citing works
Self-citationSelf-regulated Learning Processes in Secondary Education: A Network Analysis of Trace-based Measures
Cheng, Y., Guan, R., Li, T., Raković, M., Li, X., Fan, Y., Jin, F. J.-Y., Tsai, Y., Gašević, D., & Swiecki, Z. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706502 · Cited by 12 · see citing works
Self-citationvan Berk, B., Kroehne, U., & Dignath, C. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1388202 · Cited by 5 · see citing works
The lack of generalisability in learning analytics research: why, how does it matter, and where to?
Khalil, M., & Prinsloo, P. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706489 · Cited by 5 · see citing works
Alnashiri, H., Raković, M., Nawaz, S., Li, X., Lämsä, J., Lim, L., Bannert, M., Järvelä, S., & Gašević, D. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70114 · Cited by 4 · see citing works
Self-citationAnalytics of Temporal Patterns of Self-regulated Learners: A Time Series Approach
Alghamdi, S. R., Raković, M., Yang, K., Fan, Y., Gašević, D., & Chen, G. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706504 · Cited by 4 · see citing works
Self-citationLi, T., Lin, J., Iqbal, S., Swiecki, Z., Tsai, Y., Fan, Y., & Gašević, D. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47014-1_1 · Cited by 4 · see citing works
Self-citationZhang, Q., & Liang, Y. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-14038-5 · Cited by 4 · see citing works
Gray, S. L., & Cukurova, M. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2420416 · Cited by 1 · see citing works
Evaluación de habilidades metacognitivas según acciones en programación
Vincenzi, R. G. (2026) · Actualidades en Psicología · doi:10.15517/ap.v40i140.64022 · Cited by 1 · see citing works
Mirror Agents: A Dynamic Metacognitive Multi-Agent Framework for Real-Time Metacognitive Development
Liu, J., Liu, R., Wang, C., Wang, P., & Gu, X. (2025) · Proceedings. · doi:10.22318/icls2025.478580
Aksela, O., Lämsä, J., & Järvelä, S. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2666284
Ali, I. M., & Arpacı, İ. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-14087-w
Alnashiri, H., Raković, M., Nawaz, S., Li, X., Fernandez-Nieto, G. M., Qian, K., Lämsä, J., Lim, L., Bannert, M., Järvelä, S., et al. (2026) · Journal of Computer Assisted Learning · doi:10.1002/jcal.70331
Self-citationAthavale, S., Lim, L., & Bannert, M. (2026) · Metacognition and Learning · doi:10.1007/s11409-026-09500-1
Self-citation
- Authors
- Rakovic, M., Fan, Y., van der Graaf, J., Singh, S., Kilgour, J., Lim, L., Moore, J., Bannert, M., Molenaar, I., & Gasevic, D.
- Affiliations
- Centre for Learning Analytics, Faculty of Information Technology, Monash University, Australia
- University of Edinburgh (GB)
- Radboud University, Netherlands
- Monash University (AU)
- Technical University of Munich (DE)
- Radboud University Nijmegen (NL)
- Monash University, Australia
- Venue
- LAK22: 12th International Learning Analytics and Knowledge Conference
- Publisher
- ACM
- Published
- 2022-03-21
- Pages
- 130-141
- OpenAlex
- W4214834528
- Cited by
- 27
- References
- 31
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 8 · 2023: 4 · 2022: 1
- FWCI
- 6.85
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Learning Analytics and Knowledge (LAK) (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Funders
- ESRC (ES/S015701/1)
- NWO (NWO464.18.104)
- DFG (BA20144/10-1)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- metacognition, metacognitive monitoring, metacognitive regulation, automated essay scoring, essay quality, student writing support, digital learning environments, university students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
