A multimodal approach to support teacher, researcher and AI collaboration in STEM +C learning environments
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Recent advances in generative artificial intelligence (AI) and multimodal learning analytics (MMLA) have allowed for new and creative ways of leveraging AI to support K12 students' collaborative learning in STEM+C domains. To date, there is little evidence of AI methods supporting students' collaboration in complex, open‐ended environments. AI systems are known to underperform humans in (1) interpreting students' emotions in learning contexts, (2) grasping the nuances of social interactions and (3) understanding domain‐specific information that was not well‐represented in the training data. As such, combined human and AI (ie, hybrid) approaches are needed to overcome the current limitations of AI systems. In this paper, we take a first step towards investigating how a human‐AI collaboration between teachers and researchers using an AI‐generated multimodal timeline can guide and support teachers' feedback while addressing students' STEM+C difficulties as they work collaboratively to build computational models and solve problems. In doing so, we present a framework characterizing the human component of our human‐AI partnership as a collaboration between teachers and researchers. To evaluate our approach, we present our timeline to a high school teacher and discuss the key insights gleaned from our discussions. Our case study analysis reveals the effectiveness of an iterative approach to using human‐AI collaboration to address students' STEM+C challenges: the teacher can use the AI‐generated timeline to guide formative feedback for students, and the researchers can leverage the teacher's feedback to help improve the multimodal timeline. Additionally, we characterize our findings with respect to two events of interest to the teacher: (1) when the students cross a difficulty threshold, and (2) the point of intervention , that is, when the teacher (or system) should intervene to provide effective feedback. It is important to note that the teacher explained that there should be a lag between (1) and (2) to give students a chance to resolve their own difficulties. Typically, such a lag is not implemented in computer‐based learning environments that provide feedback. Practitioner notes: What is already known about this topic Collaborative, open‐ended learning environments enhance students' STEM+C conceptual understanding and practice, but they introduce additional complexities when students learn concepts spanning multiple domains. Recent advances in generative AI and MMLA allow for integrating multiple datastreams to derive holistic views of students' states, which can support more informed feedback mechanisms to address students' difficulties in complex STEM+C environments. Hybrid human‐AI approaches can help address collaborating students' STEM+C difficulties by combining the domain knowledge, emotional intelligence and social awareness of human experts with the general knowledge and efficiency of AI. What this paper adds We extend a previous human‐AI collaboration framework using a hybrid intelligence approach to characterize the human component of the partnership as a researcher‐teacher partnership and present our approach as a teacher‐researcher‐AI collaboration. We adapt an AI‐generated multimodal timeline to actualize our human‐AI collaboration by pairing the timeline with videos of students encountering difficulties, engaging in active discussions with a high school teacher while watching the videos to discern the timeline's utility in the classroom. From our discussions with the teacher, we define two types of inflection points to address students' STEM+C difficulties—the difficulty threshold and the intervention point —and discuss how the feedback latency interval separating them can inform educator interventions. We discuss two ways in which our teacher‐researcher‐AI collaboration can help teachers support students encountering STEM+C difficulties: (1) teachers using the multimodal timeline to guide feedback for students, and (2) researchers using teachers' input to iteratively refine the multimodal timeline. Implications for practice and/or policy Our case study suggests that timeline gaps (ie, disengaged behaviour identified by off‐screen students, pauses in discourse and lulls in environment actions) are particularly important for identifying inflection points and formulating formative feedback. Human‐AI collaboration exists on a dynamic spectrum and requires varying degrees of human control and AI automation depending on the context of the learning task and students' work in the environment. Our analysis of this human‐AI collaboration using a multimodal timeline can be extended in the future to support students and teachers in additional ways, for example, designing pedagogical agents that interact directly with students, developing intervention and reflection tools for teachers, helping teachers craft daily lesson plans and aiding teachers and administrators in designing curricula.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Becoming a Self-Regulated Learner: An Overview
Zimmerman, B. J. (2002) · Theory Into Practice · doi:10.1207/s15430421tip4102_2 · Cited by 10,197
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Doing Qualitative Research in Education Settings
(2002) · doi:10.1353/book4583 · Cited by 3,502
POSNER, J., RUSSELL, J. A., & PETERSON, B. S. (2005) · Development and Psychopathology · doi:10.1017/s0954579405050340 · Cited by 2,602
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Research–Practice Partnerships in Education: Outcomes, Dynamics, and Open Questions
Coburn, C. E., & Penuel, W. R. (2016) · Educational Researcher · doi:10.3102/0013189x16631750 · Cited by 939
Kapur, M. (2008) · Cognition and Instruction · doi:10.1080/07370000802212669 · Cited by 867
Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415
Human and artificial intelligence collaboration for socially shared regulation in learning
Järvelä, S., Nguyen, A., & Hadwin, A. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13325 · Cited by 342
Cukurova, M. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13514 · Cited by 302
Cited by
Ali, M., Ma, M., Muneeb, M., & Wong, G. K. W. (2025) · Metrics · doi:10.3390/metrics2040023 · Cited by 6
Du, J. (., & Kohnke, L. (2026) · Teaching Education · doi:10.1080/10476210.2025.2590694 · Cited by 4
Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents
Cohn, C., Guo, S., Rayala, S., Wang, H. D., Mohammed, N., Timalsina, U., Jain, S., Eeds, A., Deweese, M., Osborn Popp, P. J., et al. (2027) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-29744-0_1
“Over four years of participatory design (PD) with students, teachers, and learning scientists, we synthesized surveys, interviews, focus groups, pilot studies, and collaborative design activities into a framework aligned with established theory [10,5,9,8,11,13].”
Self-citation
Metadata
- Authors
- Cohn, C., Snyder, C., Fonteles, J. H., T. S., A., Montenegro, J., & Biswas, G.
- Venue
- British Journal of Educational Technology
- Publisher
- Wiley
- Published
- 2024-09-18
- Volume
- 56
- Issue
- 2
- Pages
- 595-620
- ISSN
- 0007-1013 (ISSN-L)
- 1467-8535
- OpenAlex
- W4402606299
- Cited by
- 65
- References
- 13
- FWCI
- 9.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.