CoChat
Sign Up
Cited by 124Online Learning and AnalyticsInnovative Teaching and Learning MethodsIntelligent Tutoring Systems and Adaptive Learning

An artificial intelligence-driven learning analytics method to examine the collaborative problem-solving process from the complex adaptive systems perspective

Open access
Ouyang, Fan, Xu, Weiqi, Cukurova, Mutlu·2023·International Journal of Computer-Supported Collaborative Learning
DOI 10.1007/s11412-023-09387-z
Open this paper in CoChat — free accountOpen PDF

Abstract

Collaborative problem solving (CPS) enables student groups to complete learning tasks, construct knowledge, and solve problems. Previous research has argued the importance of examining the complexity of CPS, including its multimodality, dynamics, and synergy from the complex adaptive systems perspective. However, there is limited empirical research examining the adaptive and temporal characteristics of CPS, which may have led to an oversimplified representation of the real complexity of the CPS process. To expand our understanding of the nature of CPS in online interaction settings, the present research collected multimodal process and performance data (i.e., speech, computer screen recordings, concept map data) and proposed a three-layered analytical framework that integrated AI algorithms with learning analytics to analyze the regularity of groups’ collaboration patterns. The results surfaced three types of collaborative patterns in groups, namely the behaviour-oriented collaborative pattern (Type 1) associated with medium-level performance, the communication-behaviour-synergistic collaborative pattern (Type 2) associated with high-level performance, and the communication-oriented collaborative pattern (Type 3) associated with low-level performance. This research further highlighted the multimodal, dynamic, and synergistic characteristics of groups’ collaborative patterns to explain the emergence of an adaptive, self-organizing system during the CPS process. According to the empirical research results, theoretical, pedagogical, and analytical implications were discussed to guide the future research and practice of CPS.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11412-023-09387-z

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 74

  1. Abbott, A., & Tsay, A. (2000). Sequence analysis and optimal matching methods in sociology: review and prospect. Sociological Methods & Research, 29(1), 3-33. https://doi.org/10.1177/0049124100029001 001. · DOI
  2. Amon, M. J., Vrzakova, H., & D'Mello, S. K. (2019). Beyond dyadic coordination: Multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cognitive Science, 43(10), https:// doi.org/10.1111/cogs.12787. Article e12787. · DOI
  3. Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9(4), 403-436. https://doi.org/10.1207/S15327809JLS0904_2. · DOI
  4. Blikstein, P. (2013). Multimodal learning analytics. In R. F. Kizilcec, C. Piech, E. Schneider, D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the third international conference on learning analytics and knowledge (pp. 102-106). ACM. https://doi.org/10.1145/2460296.2460316 · DOI
  5. Borge, M., & Mercier, E. (2019). Towards a micro-ecological approach to CSCL. International Jour- nal of Computer-Supported Collaborative Learning, 14(2), 219-235. https://doi.org/10.1007/ s11412-019-09301-6. · DOI
  6. Byrne, D., & Callaghan, G. (2014). Complexity theory and the social sciences. Routledge.
  7. Cress, U., Rosé, C., Wise, A., & Oshima, J. (2021). International handbook of computer-supported collab- orative learning. Springer.
  8. Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multi- modal learning analytics. British Journal of Educational Technology, 51(5), 1441-1449. https://doi. org/10.1111/bjet.13015. · DOI · Paper page
  9. Curşeu, P. L., Rusu, A., Maricuţoiu, L. P., Vîrgă, D., & Măgurean, S. (2020). Identified and engaged: a multi- level dynamic model of identification with the group and performance in collaborative learning. Learn- ing and Individual Differences, 78, 101838. https://doi.org/10.1016/j.lindif.2020.101838. · DOI
  10. Damşa, C. I. (2014). The multi-layered nature of small-group learning: productive interactions in object-ori- ented collaboration. International Journal of Computer-Supported Collaborative Learning, 9, 247-281. https://doi.org/10.1007/s11412-014-9193-8. · DOI
  11. de Carvalho, W. F., & Zárate, L. E. (2020). A new local causal learning algorithm applied in learning analyt- ics. The International Journal of Information and Learning Technology, 38(1), 103-115. https://doi. org/10.1108/IJILT-04-2020-0046. · DOI
  12. Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative- learning: cognitive and computational approaches (pp. 1-19). Elsevier.
  13. Eddy, S. R. (1996). Hidden Markov models. Current Opinion in Structural Biology, 6(3), 361-365. https:// doi.org/10.1016/S0959-440X(96)80056-X. · DOI
  14. Eisenberg-Guyot, J., Peckham, T., Andrea, S. B., Oddo, V., Seixas, N., & Hajat, A. (2020). Life-course tra- jectories of employment quality and health in the US: a multichannel sequence analysis. Social Science & Medicine, 264, 113327. https://doi.org/10.1016/j.socscimed.2020.113327. · DOI
  15. Eloy, L., Stewart, E. B., Jean Amon, A., Reinhardt, M., Michaels, C., Sun, A., & D'Mello, C. (2019). S. Modeling team-level multimodal dynamics during multiparty collaboration. In W. Gao, H. M. L. Meng, M. Turk. R. Fussell, B. Schuller, Y. Song, & K. Yu (Eds.), 2019 International conference on multimodal interaction (pp. 244-258). ACM. https://doi.org/10.1145/3340555.3353748 · DOI
  16. Farrokhnia, M., Pijeira-Díaz, H. J., Noroozi, O., & Hatami, J. (2019). Computer-supported collabora- tive concept mapping: the effects of different instructional designs on conceptual understanding and knowledge co-construction. Computers & Education, 142, Article 103640. https://doi.org/10.1016/j. compedu.2019.103640. · DOI
  17. Felsenstein, J., & Churchill, G. A. (1996). A hidden Markov Model approach to variation among sites in rate of evolution. Molecular Biology and Evolution, 13(1), 93-104. https://doi.org/10.1093/oxfordjournals. molbev.a025575. · DOI
  18. Fine, S., Singer, Y., & Tishby, N. (1998). The hierarchical hidden Markov model: analysis and applications. Machine Learning, 32, 41-62. https://doi.org/10.1023/A:1007469218079. · DOI · Paper page
  19. Fiore, S. M., Smith-Jentsch, K. A., Salas, E., Warner, N., & Letsky, M. (2010). Toward an understanding of macrocognition in teams: developing and defining complex collaborative processes and products. Theo- retical Issues in Ergonomic Science, 11(4), 250-271. https://doi.org/10.1080/14639221003729128. · DOI
  20. Gabadinho, A., Ritschard, G., Müller, N. S., & Studer, M. (2011). Analyzing and visualizing state sequences in R with TraMineR. Journal of Statistical Software, 40(4), 1-37. https://doi.org/10.18637/jss.v040.i04. · DOI
  21. Gauthier, J. A., Widmer, E. D., Bucher, P., & Notredame, C. (2010). Multichannel sequence analysis applied to social data. Sociological Methodology, 40(1), 1-38. https://doi.org/10.1111/J.1467-9531.2010.01227. · DOI
  22. X. · DOI
  23. Gorman, J. C., Grimm, D. A., Stevens, R. H., Galloway, T., Willemsen-Dunlap, A. M., & Halpin, D. J. (2020). Measuring real-time team cognition during team training. Human Factors, 62(5), 825-860. https://doi. org/10.1177/0018720819852791. · DOI
  24. Helske, S., & Helske, J. (2019). Mixture hidden Markov models for sequence data: the seqHMM Package in R. Journal of Statistical Software, 88(3), 1-32. https://doi.org/10.18637/jss.v088.i03. · DOI
  25. Hmelo-Silver, C. E. (2004). Problem-based learning: what and how do students learn? Educational Psychol- ogy Review, 16, 235-266. https://doi.org/10.1023/B:EDPR.0000034022.16470. · DOI
  26. Hmelo-Silver, C. E., & DeSimone, C. (2013). Problem-based learning: an instructional model of collab- orative learning. In C. Hmelo-Silver, C. A. Chinn, C. Chan, & A. O'Donnell (Eds.), The international handbook of collaborative learning. Routledge.
  27. Hoppe, H. U., Doberstein, D., & Hecking, T. (2021). Using sequence analysis to determine the well-function- ing of small groups in large online courses. International Journal of Artificial Intelligence in Education, 31, 680-699. https://doi.org/10.1007/s40593-020-00229-9. · DOI
  28. Jacobson, M. J., Kapur, M., & Reimann, P. (2016). Conceptualizing debates in learning and educational research: toward a complex systems conceptual framework of learning. Educational Psychologist, 51(2), 210-218. https://doi.org/10.1080/00461520.2016.1166963. · DOI
  29. Janssen, J., Cress, U., Erkens, G., & Kirschner, P. A. (2013). Multilevel analysis for the analysis of collab- orative learning. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan, & A. M. O'Donnell (Eds.), The international handbook of collaborative learning (pp. 124-137). Routledge.
  30. Jeong, H. (2013). Verbal data analysis for understanding interactions. In C. E. Hmelo-Silver, C. K. K. Chan, C. Chinn, & A. M. O'Donnell (Eds.), International handbook of collaborative learning (pp. 168-181). Routledge.
  31. Kaendler, C., Wiedmann, M., Rummel, N., & Spada, H. (2015). Teacher competencies for the implementa- tion of collaborative learning in the classroom: a framework and research. Educational Psychology Review, 27(3), 1-32. https://doi.org/10.1007/s10648-014-9288-9. · DOI
  32. Khan, S. M. (2017). Multimodal behavioral analytics in intelligent learning and assessment systems. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative Assessment of Collaboration (pp. 173-184). Springer. https://doi.org/10.1007/978-3-319-33261-1_11 · DOI
  33. Koopmans, M., & Stamovlasis, D. (2016). Complex dynamical systems in education. Springer International Publishing.
  34. Kuhn, D., Capon, N., & Lai, H. (2020). Talking about group (but not individual) process aids group perfor- mance. International Journal of Computer-Supported Collaborative Learning, 15, 179-192. https://doi. org/10.1007/s11412-020-09321-7. · DOI
  35. Kwon, K., Liu, Y. H., & Johnson, L. P. (2014). Group regulation and social-emotional interactions observed in computer supported collaborative learning: comparison between good vs. poor collaborators. Com- puters & Education, 78, 185-200. https://doi.org/10.1016/j.compedu.2014.06.004. · DOI
  36. Lansing, J. S. (2003). Complex adaptive systems. Annual Review of Anthropology, 32(1), 183-204. https:// doi.org/10.1146/annurev.anthro.32.061002.093440. · DOI
  37. Malmberg, J., Järvelä, S., & Järvenoja, H. (2017). Capturing temporal and sequential patterns of self-, co-, and socially shared regulation in the context of collaborative learning. Contemporary Educational Psy- chology, 49, 160-174. https://doi.org/10.1016/j.cedpsych.2017.01.009. · DOI
  38. Medina, R., & Stahl, G. (2021). Analysis of group practices. In U. Cress, C. Rosé, A. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 199-218). Springer.
  39. Mitchell, M. (2009). Complexity: a guided tour. Oxford University Press.
  40. Morrison, K. R. B. (2002). School leadership and complexity theory. Routledge.
  41. Mu, S., Cui, M., & Huang, X. (2020). Multimodal data fusion in learning analytics: a systematic review. Sen- sors (Basel, Switzerland), 20(23), 6856, 1-26. https://doi.org/10.3390/s20236856. · DOI
  42. Murtagh, F., & Legendre, P. (2014). Ward's hierarchical agglomerative clustering method: which algo- rithms implement Ward's criterion? Journal of Classification, 31(3), 274-295. https://doi.org/10.1007/ s00357-014-9161-z. · DOI
  43. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan, & A. M. O'Donnell (Eds.), The international hand- book of collaborative learning (pp. 93-111). Routledge.
  44. Olsen, J. K., Sharma, K., Rummel, N., & Aleven, V. (2020). Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology, 51(5), 1527-1547. https:// doi.org/10.1111/bjet.12982. · DOI
  45. Ouyang, F. (2021). Using three social network analysis approaches to understand computer-supported collaborative learning. Journal of Educational Computing Research, 59(7), 1401-1424. https://doi. org/10.1177/0735633121996477. · DOI
  46. Ouyang, F., & Chang, Y. H. (2019). The relationships between social participatory roles and cognitive engagement levels in online discussions. British Journal of Educational Technology, 50(3), 1396-1414. https://doi.org/10.1111/bjet.12647. · DOI
  47. Ouyang, F., Chen, Z., Cheng, M., Tang, Z., & Su, C. Y. (2021). Exploring the effect of three scaffoldings on the collaborative problem-solving processes in China's higher education. International Journal of Edu- cational Technology in Higher Education, 18(35), 1-22. https://doi.org/10.1186/s41239-021-00273-y. · DOI
  48. Ouyang, F., Dai, X., & Chen, S. (2022). Applying multimodal learning analytics to examine the immediate and delayed effects of instructor scaffoldings on small groups' collaborative programming. Interna- tional Journal of STEM Education, 9(1), 1-21. https://doi.org/10.1186/s40594-022-00361-z. · DOI
  49. Ouyang, F., & Scharber, C. (2017). The influences of an experienced instructor's discussion design and facili- tation on an online learning community development: a social network analysis study. The Internet and Higher Education, 35, 34-47. https://doi.org/10.1016/j.iheduc.2017.07.002. · DOI
  50. Ouyang, F., Wu, M., Zhang, L., Xu, W., Zheng, L., & Cukurova, M. (2023). Making strides towards AI-sup- ported regulation of learning in collaborative knowledge construction. Computers in Human Behavior, 142, 107650. https://doi.org/10.1016/j.chb.2023.107650. · DOI
  51. Ouyang, F., & Xu, W. (2022). The effects of three instructor participatory roles on a small group's collab- orative concept mapping. Journal of Educational Computing Research, 60(4), 930-959. https://doi. org/10.1177/07356331211057283. · DOI
  52. Park, J. B. H., Schallert, D. L., Sanders, A. J. Z., Williams, K. M., Seo, E., Yu, L. T., Vogler, J. S., Song, K., Williamson, Z. H., & Knox, M. C. (2015). Does it matter if the teacher is there? A teacher's contribu- tion to emerging patterns of interactions in online classroom discussions. Computers & Education, 82, 315-328. https://doi.org/10.1016/j.compedu.2014.11.019. · DOI
  53. Ramenzoni, V. C., Riley, M. A., Shockley, K., & Baker, A. A. (2012). Interpersonal and intrapersonal coor- dinative modes for joint and single task performance. Human Movement Science, 31(5), 1253-1267. https://doi.org/10.1016/j.humov.2011.12.004. · DOI
  54. Rogat, T. K., & Adams-Wiggins, K. R. (2015). Interrelation between regulatory and socioemotional pro- cesses within collaborative groups characterized by facilitative and directive other-regulation. Comput- ers in Human Behavior, 52, 589-600. https://doi.org/10.1016/j.chb.2015.01.026. · DOI
  55. Saqr, M., & López-Pernas, S. (2022). How CSCL roles emerge, persist, transition, and evolve over time: a four- year longitudinal study. Computers & Education, 189, https://doi.org/10.1016/j.compedu.2022.104581. Article 104581. · DOI
  56. Saqr, M., Peeters, W., & Viberg, O. (2021). The relational, co-temporal, contemporaneous, and longitudi- nal dynamics of self-regulation for academic writing. Research and Practice in Technology Enhanced Learning, 16(1), 1-22. https://doi.org/10.1186/s41039-021-00175-7Schindler, M., & Bakker, A. (2020). Affective field during collaborative problem posing and problem solving: A case study. Educational Studies in Mathematics, 105(3), 303-324. https://doi.org/10.1007/s10649-020-09973-0 · DOI
  57. Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9-45. https://doi.org/10.18608/jla.2016.33.3. Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: what can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450-1484. https://doi. org/10.1111/bjet.12993. · DOI
  58. Stahl, G. (2009). Studying virtual math teams. Springer.
  59. Stahl, G. (2013). Learning across levels. International Journal of Computer-Supported Collaborative Learn- ing, 8(1), 1-12. https://doi.org/10.1007/s11412-013-9169-0. · DOI
  60. Stahl, G. (2017). Group practices: A new way of viewing CSCL. International Journal of Computer-Sup- ported Collaborative Learning, 12(1), 113-126.10.1007/s11412-017-9251-0 · DOI
  61. Stahl, G., & Hakkarainen, K. (2021). Theories of CSCL. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 23-44). Springer. https:// doi.org/10.1007/978-3-030-65291-3_2 · DOI
  62. Sullivan, F. R., & Keith, P. K. (2019). Exploring the potential of natural language processing to support microgenetic analysis of collaborative learning discussions. British Journal of Educational Technology, 50(6), 3047-3063. https://doi.org/10.1111/bjet.12875. · DOI
  63. Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2015). Teacher regulation of cognitive activi- ties during student collaboration: Effects of learning analytics. Computers & Education, 90(1), 80-94. https://doi.org/10.1016/j.compedu.2015.09.006. · DOI
  64. Vogler, J. S., Schallert, D. L., Jordan, M. E., Song, K., Sanders, A. J., Chiang, T., & Yu, Y. H. Y., L. T (2017). Life history of a topic in an online discussion: a complex systems theory perspective on how one message attracts class members to create meaning collaboratively. International Journal of Computer- Supported Collaborative Learning, 12(2), 173-194. https://doi.org/10.1007/s11412-017-9255-9. · DOI
  65. Vrzakova, H., Amon, M. J., Stewart, A., Duran, N. D., & D'Mello, S. K. (2020). Focused or stuck together: Multimodal patterns reveal triads' performance in collaborative problem solving. In C. Rensing, & H. Drachsler (Eds.), Proceedings of the tenth international conference on learning analytics & knowledge (pp. 295-304). ACM. https://doi.org/10.1145/3375462.3375467 · DOI
  66. Vygotsky, L. S. (1978). Mind in society: the development of higher psychological processes. Harvard Uni- versity Press.
  67. Wang, M., Cheng, B., Chen, J., Mercer, N., & Kirschner, P. A. (2017). The use of web-based collaborative concept mapping to support group learning and interaction in an online environment. Internet and Higher Education, 34, 28-40. https://doi.org/10.1016/j.iheduc.2017.04.003. · DOI
  68. Wiltshire, T. J., Steffensen, S. V., & Fiore, S. M. (2019). Multiscale movement coordination dynamics in collaborative team problem solving. Applied Ergonomics, 79, 143-151. https://doi.org/10.1016/j. apergo.2018.07.007. · DOI
  69. Wise, A. F., Knight, S., & Shum, S. B. (2021). Collaborative learning analytics. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 425-444). Springer. https://doi.org/10.1007/978-3-030-65291-3_23 · DOI
  70. Zemel, A., & Koschmann, T. (2013). Recalibrating reference within a dual-space interaction environ- ment. International Journal of Computer-Supported Collaborative Learning, 8(1), 65-87. https://doi. org/10.1007/s11412-013-9164-5. · DOI
  71. Zhang, J. (2007). A cultural look at information and communication technologies in eastern educa- tion. Educational Technology Research and Development, 55(3), 301-314. https://doi.org/10.1007/ s11423-007-9040-y. · DOI
  72. Zhang, J. (2013). Chapter 28: collaboration, technology, and culture. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan, & A. M. O'Donnell (Eds.), The international handbook of collaborative learning (pp. 495-508). New York, NY: Routledge.
  73. Zheng, L., Zhong, L., Niu, J., Long, M., & Zhao, J. (2021). Effects of personalized intervention on collabora- tive knowledge building, group performance, socially shared metacognitive regulation, and cognitive load in computer-supported collaborative learning. Educational Technology & Society, 24(3), 174-193. https://www.jstor.org/stable/27032864.
  74. Zuiker, S. J., Anderson, K. T., Jordan, M. E., & Stewart, O. G. (2016). Complementary lenses: using theo- ries of situativity and complexity to understand collaborative learning as systems-level social activity. Learning Culture and Social Interaction, 9, 80-94. https://doi.org/10.1016/J.LCSI.2016.02.003. · DOI

Cited by

Sorted by citations

Authors
Ouyang, F., Xu, W., & Cukurova, M.
Affiliations
  • Zhejiang University (CN)
  • University College London (GB)
Corresponding author
  • Ouyang, Fan
  • Cukurova, Mutlu
ORCID
Venue
International Journal of Computer-Supported Collaborative Learning
Publisher
Springer Science and Business Media LLC
Published
2023-03
Volume
18
Issue
1
Pages
39-66
ISSN
  • 1556-1607 (ISSN-L)
  • 1556-1615
DOI
10.1007/s11412-023-09387-z
OpenAlex
W4360985445
Cited by
124
References
7
Citations by year
2026: 35 · 2025: 37 · 2024: 41 · 2023: 11
FWCI
28.27
Type
journal article
Language
en
Update notices
correction (2023-04-10)
Open access
Licence
Locations
International Journal of Computer-Supported Collaborative Learning (publishedVersion, cc-by)
Funders
National Natural Science Foundation of China (62177041)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
collaborative problem solving, complex adaptive systems, learning analytics, multimodal data, speech analysis, screen recording, concept maps, group performance, self-organization, online collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review