Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
We investigated the generalizability of language-based analytics models across two collaborative problem solving (CPS) tasks: an educational physics game and a block programming challenge. We analyzed a dataset of 95 triads (N=285) who used videoconferencing to collaborate on both tasks for an hour. We trained supervised natural language processing classifiers on automatic speech recognition transcripts to predict the human-coded CPS facets (skills) of constructing shared knowledge, negotiation / coordination, and maintaining team function. We tested three methods for representing collaborative discourse: (1) deep transfer learning (using BERT), (2) n-grams (counts of words/phrases), and (3) word categories (using the Linguistic Inquiry Word Count [LIWC] dictionary). We found that the BERT and LIWC methods generalized across tasks with only a small degradation in performance (Transfer Ratio of .93 with 1 indicating perfect transfer), while the n-grams had limited generalizability (Transfer Ratio of .86), suggesting overfitting to task-specific language. We discuss the implications of our findings for deploying language-based collaboration analytics in authentic educational environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
A theory of learning from different domains
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010) · Machine Learning · doi:10.1007/s10994-009-5152-4 · Cited by 3,515 · cited 2×
“This experiment was informed by domain adaptation theory, which suggests that cross-domain generalization can be achieved through feature representations from which the domain of an input example cannot be identified [3].”
MethodResultThe Construction of Shared Knowledge in Collaborative Problem Solving.
Roschelle, J., & Teasley, S. D. (1995) · Computer Supported Collaborative Learning · doi:10.1007/978-3-642-85098-1_5 · Cited by 2,274 · cited 1×
“Collaborative problem solving (CPS) - defined as two or more people engaging in a coordinated attempt to construct and maintain a joint solution to a problem [39] - is considered a critical skill for the”
BackgroundRosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-007-9034-0 · Cited by 351
Towards a generalized competency model of collaborative problem solving
Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D'Mello, S. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103672 · Cited by 189
Self-citationCollaborative Problem-Solving Education for the Twenty-First-Century Workforce
Fiore, S. M., Graesser, A., & Greiff, S. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0363-y · Cited by 113 · cited 3×
“Accordingly, educational researchers, practitioners, and policymakers across the globe have emphasized the importance of improving and expanding CPS instruction in our education systems [10, 16, 17, 20, 31].”
BackgroundMultimodal modeling of collaborative problem-solving facets in triads
Stewart, A. E. B., Keirn, Z., & D’Mello, S. K. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09290-y · Cited by 30 · cited 1×
“…data (either gathered from text chats or transcribed from speech) has been used to model important CPS skills such as negotiation [18, 23, 36, 51, 52], information sharing [18, 23, 36, 51], regulation [15] and argumentation [6, 40], in addition to predicting CPS outcomes such as learning gains…”
MethodSelf-citation
Cited by
Multimodal large language models for inclusive collaboration learning tasks
Lewis, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop · doi:10.18653/v1/2022.naacl-srw.26 · Cited by 30
“A Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse.”
Cao, J., Ganesh, A., Cai, J., Southwell, R., Perkoff, E. M., Regan, M., Kann, K., Martin, J. H., Palmer, M., & D'Mello, S. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3595606 · Cited by 25
Self-citationD’Mello, S. K., Duran, N., Michaels, A., & Stewart, A. E. B. (2024) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09387-6 · Cited by 14
Self-citationChandler, C., Breideband, T., Reitman, J. G., Chitwood, M., Bush, J. B., Howard, A., Leonhart, S., Foltz, P. W., Penuel, W. R., & D'Mello, S. K. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636917 · Cited by 8
Self-citation
Metadata
- Authors
- Pugh, S. L., Rao, A., Stewart, A. E. B., & D'Mello, S. K.
- Venue
- LAK22: 12th International Learning Analytics and Knowledge Conference
- Publisher
- ACM
- Published
- 2022-03-21
- Pages
- 208-218
- OpenAlex
- W4214844842
- Cited by
- 31
- References
- 7
- FWCI
- 10.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do Speech-Based Collaboration Analytics Generalize Across Task Contexts? https://app.cochat.ai/papers/10.1145/3506860.3506894
- The paper
- Pugh, S. L., Rao, A., Stewart, A. E. B., & D'Mello, S. K. (2022). Do Speech-Based Collaboration Analytics Generalize Across Task Contexts? LAK22: 12th International Learning Analytics and Knowledge Conference. https://doi.org/10.1145/3506860.3506894
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.