CoChat
Sign Up
Cited by 47Intelligent Tutoring Systems and Adaptive LearningInnovative Teaching and Learning MethodsArtificial Intelligence in Education

Pair-Up: Prototyping Human-AI Co-orchestration of Dynamic Transitions between Individual and Collaborative Learning in the Classroom

Open access
Yang, Kexin Bella, Echeverria, Vanessa, Lu, Zijing, Mao, Hongyu, Holstein, Kenneth, Rummel, Nikol, Aleven, Vincent·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3581398
Open this paper in CoChat — free accountOpen PDF

Abstract

Enabling students to dynamically transition between individual and collaborative learning activities has great potential to support better learning. We explore how technology can support teachers in orchestrating dynamic transitions during class. Working with five teachers and 199 students over 22 class sessions, we conducted classroom-based prototyping of a co-orchestration technology ecosystem that supports the dynamic pairing of students working with intelligent tutoring systems. Using mixed-methods data analysis, we study the resulting observed classroom dynamics, and how teachers and students perceived and experienced dynamic transitions as supported by our technology. We discover a potential tension between teachers’ and students’ preferred level of control: students prefer a degree of control over the dynamic transitions that teachers are hesitant to grant. Our study reveals design implications and challenges for future human-AI co-orchestration in classroom use, bringing us closer to realizing the vision of highly-personalized smart classrooms that address the unique needs of each student.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3581398

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Higher education students have broader expectations of independence and a preference for agency in AI-assisted classrooms.

    “This aligns with higher education students' broader expectations of independence [134], and their preference for agency in AI-assisted classrooms [139].”

    From AI Sensing and Intervention in Higher Education: Student Perceptions of Learning Impacts, Affective Responses, and Ethical Priorities (2026) · cites this paper as [139]

    Section 6.3.1 reports that students wanted some control over their partners and the timing of transitions, supporting the preference-for-agency point attributed to this citation. But section 4.1 identifies the participants as sixth- to eighth-grade students, not higher education students, so the cited study does not establish that preference for the population named. The independence point is attached to a different citation [134]. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Little is known about how people should approach and prepare for LLM-based chatbots that are being used in all areas of education.

    “While researchers in HCI and AIEd have long accumulated relevant knowledge of educational AI and even established a basis for human-AI partnership in the classroom [98] beyond the development of successful AI tutors, little is known about how people should approach and prepare for LLM-based chatbots that are being used in all areas of education.”

    From The Promise and Peril of ChatGPT in Higher Education: Opportunities, Challenges, and Design Implications (2024) · cites this paper as [98]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. Hamed Alavi and Pierre Dillenbourg. 2012. An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving. Learning Technologies, IEEE Transactions on 5 (07 2012), 264-274. https://doi.org/10.1109/TLT.2012.7 · DOI
  2. Vincent Aleven and Kenneth R Koedinger. 2013. Knowledge component (KC) approaches to learner modeling. Design Recommendations for Intelligent Tutoring Systems 1 (2013), 165-182.
  3. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human- AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3290605.3300233 · DOI · Paper page
  4. Pengcheng An, Saskia Bakker, Sara Ordanovski, Ruurd Taconis, Chris L.E. Paffen, and Berry Eggen. 2019. Unobtrusively Enhancing Reflection-in-Action of Teach- ers through Spatially Distributed Ambient Information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3290605.3300321 · DOI
  5. Bartel-undated-mj A. N. Bartel, E. M. Silla, N. A. Vest, T. Nagashima, V. Aleven, and M. W. Alibali. [n. d.]. Reasoning about equations with tape diagrams: Insights from math teachers and college students. Proceedings of the 15th International Conference of the Learning Sciences -ICLS 2021 ([n. d.]). https://doi.org/10.22318/ icls2021.685 · DOI
  6. B Bateman. 1992. Learning disabilities: the changing landscape. J. Learn. Disabil. 25, 1 (Jan. 1992), 29-36. https://doi.org/10.1177/002221949202500105 · DOI
  7. Joseph Beck and Yue Gong. 2013. Wheel-Spinning: Students Who Fail to Master a Skill, Vol. 7926. 431-440. https://doi.org/10.1007/978-3-642-39112-5_44 · DOI
  8. Matthew Bernacki, Jeffrey Greene, and Helen Crompton. 2019. Mobile Technol- ogy, Learning, and Achievement: Advances in Understanding and Measuring the Role of Mobile Technology in Education. Contemporary Educational Psychology 60 (11 2019), 101827. https://doi.org/10.1016/j.cedpsych.2019.101827 · DOI
  9. Virginia Braun and Victoria Clarke. 2016. (Mis)conceptualising themes, thematic analysis, and other problems with Fugard and Potts' (2015) sample-size tool for thematic analysis. International Journal of Social Research Methodology 19 (06 2016), 1-5. https://doi.org/10.1080/13645579.2016.1195588 · DOI
  10. Reviewed by Elmar Hashimov. 2015. Qualitative Data Analysis: A Methods Sourcebook and The Coding Manual for Qualitative Researchers. Technical Communication Quarterly 24, 1 (2015), 109-112. https://doi.org/10.1080/10572252. 2015.975966 arXiv:https://doi.org/10.1080/10572252.2015.975966 · DOI
  11. Hao Cen, Kenneth Koedinger, and Brian Junker. 2006. Learning Factors Analysis -A General Method for Cognitive Model Evaluation and Improvement, Vol. 4053. 164-175. https://doi.org/10.1007/11774303_17 · DOI · Paper page
  12. Albert T Corbett and John R Anderson. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-adapt Interact. 4, 4 (1995), 253-278. https://doi.org/10.1007/BF01099821 · DOI · Paper page
  13. Sébastien Cuendet, Quentin Bonnard, Frederic Kaplan, and Pierre Dillenbourg. 2011. Paper Interface Design for Classroom Orchestration. (05 2011). https: //doi.org/10.1145/1979742.1979904 · DOI
  14. Pierre Dillenbourg and Patrick Jermann. 2010. Technology for Classroom Orches- tration. Springer New York, New York, NY, 525-552. https://doi.org/10.1007/978- 1-4419-5716-0_26 · DOI
  15. Vanessa Echeverria, Kenneth Holstein, Jennifer Huang, Jonathan Sewall, Nikol Rummel, and Vincent Aleven. 2020. Exploring Human-AI Control Over Dynamic Transitions Between Individual and Collaborative Learning. In Addressing Global Challenges and Quality Education, Carlos Alario-Hoyos, María Jesús Rodríguez- Triana, Maren Scheffel, Inmaculada Arnedillo-Sánchez, and Sebastian Maximilian Dennerlein (Eds.). Springer International Publishing, Cham, 230-243.
  16. Kenneth Holstein, Bruce Mclaren, and Vincent Aleven. 2018. Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms. 154-168. https://doi.org/10.1007/978-3-319-93843-1_12 · DOI · Paper page
  17. Kenneth Holstein, Bruce Mclaren, and Vincent Aleven. 2019. Co-Designing a Real- Time Classroom Orchestration Tool to Support Teacher-AI Complementarity. Journal of Learning Analytics 6 (08 2019), 27-52. https://doi.org/10.18608/jla. 2019.62.3 · DOI
  18. Kenneth Holstein, Bruce M. McLaren, and Vincent Aleven. 2017. Intelligent Tutors as Teachers' Aides: Exploring Teacher Needs for Real-Time Analyt- ics in Blended Classrooms. In Proceedings of the Seventh International Learn- ing Analytics & Knowledge Conference (Vancouver, British Columbia, Canada) (LAK '17). Association for Computing Machinery, New York, NY, USA, 257-266. https://doi.org/10.1145/3027385.3027451 · DOI
  19. Stian Håklev, Louis Faucon, Thanasis Hadzilacos, and Pierre Dillenbourg. 2017. Orchestration Graphs: Enabling Rich Social Pedagogical Scenarios in MOOCs. https://doi.org/10.1145/3051457.3054000 · DOI
  20. Trevor Kerry and Carolle A. Kerry. 1997. Differentiation: teachers' views of the usefulness of recommended strategies in helping the more able pupils in primary and secondary classrooms. Educational Studies 23, 3 (1997), 439-457. https://doi. org/10.1080/0305569970230309 arXiv:https://doi.org/10.1080/0305569970230309 · DOI
  21. Ahmed Kharrufa, Sally Rix, Timur Osadchiy, Anne Preston, and Patrick Olivier. 2017. Group Spinner: Recognizing and Visualizing Learning in the Classroom for Reflection, Communication, and Planning. 5556-5567. https://doi.org/10.1145/ 3025453.3025679 · DOI
  22. Aditi Kothiyal, Sahana Murthy, and Sridhar Iyer. 2014. Think-pair-share in a large CS1 class: Does learning really happen? ITICSE 2014 -Proceedings of the 2014 Innovation and Technology in Computer Science Education Conference (06 2014). https://doi.org/10.1145/2591708.2591739 · DOI
  23. Chun-Yu Lin, Chung-Kai Huang, and Chao-Jung Ko. 2020. The impact of per- ceived enjoyment on team effectiveness and individual learning in a blended learning business course: The mediating effect of knowledge sharing. Aus- tralasian Journal of Educational Technology 36 (01 2020), 126-141. https: //doi.org/10.14742/ajet.4446 · DOI
  24. Yanjin Long and Vincent Aleven. 2013. Supporting Students' Self-Regulated Learning with an Open Learner Model in a Linear Equation Tutor, Vol. 7926. https://doi.org/10.1007/978-3-642-39112-5_23 · DOI
  25. Roberto Martinez-Maldonado, Andrew Clayphan, and Judy Kay. 2015. Deploying and Visualising Teacher's Scripts of Small Group Activities in a Multi-surface Classroom Ecology: a Study in-the-wild. Computer Supported Cooperative Work (CSCW) 24 (06 2015). https://doi.org/10.1007/s10606-015-9217-6 · DOI
  26. Roberto Martinez-Maldonado, Andrew Clayphan, Kalina Yacef, and Judy Kay. 2015. MTFeedback: Providing Notifications to Enhance Teacher Awareness of Small Group Work in the Classroom. Learning Technologies, IEEE Transactions on 8 (04 2015), 187-200. https://doi.org/10.1109/TLT.2014.2365027 · DOI
  27. Roberto Martinez-Maldonado, Yannis Dimitriadis, Judy Kay, Kalina Yacef, and Marie-Theresa Edbauer. 2013. MTClassroom and MTDashboard: Supporting analysis of teacher attention in an orchestrated multi-tabletop classroom. Proc. CSCL2013 1, 119-128.
  28. Emma Mercier. 2016. Teacher orchestration and student learning during mathe- matics activities in a smart classroom. International Journal of Smart Technology and Learning 1 (01 2016), 33. https://doi.org/10.1504/IJSMARTTL.2016.078160 · DOI
  29. Inge Molenaar, Anne Horvers, and Ryan Baker. 2019. Towards Hybrid Human- System Regulation: Understanding Children' SRL Support Needs in Blended Classrooms. LAK19: Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 471-480. https://doi.org/10.1145/3303772.3303780 · DOI
  30. Hannelore Montrieux, Ruben Vanderlinde, Tammy Schellens, and Lieven Marez. 2015. Teaching and Learning with Mobile Technology: A Qualitative Explorative Study about the Introduction of Tablet Devices in Secondary Education. PloS one 10 (12 2015), e0144008. https://doi.org/10.1371/journal.pone.0144008 · DOI
  31. Dejana Mullins, Nikol Rummel, and Hans Spada. 2011. Are two heads always better than one? Differential effects of collaboration on students' computer- supported learning in mathematics. I. J. Computer-Supported Collaborative Learn- ing 6 (09 2011), 421-443. https://doi.org/10.1007/s11412-011-9122-z · DOI
  32. Jennifer Olsen, Nikol Rummel, and Vincent Aleven. 2019. It is not either or: An initial investigation into combining collaborative and individual learning using an ITS. International Journal of Computer-Supported Collaborative Learning 14 (10 2019), 1-29. https://doi.org/10.1007/s11412-019-09307-0 · DOI
  33. Jennifer Olsen, Nikol Rummel, and Vincent Aleven. 2020. Designing for the co- Orchestration of Social Transitions between Individual, Small-Group and Whole- Class Learning in the Classroom. International Journal of Artificial Intelligence in Education 31 (10 2020). https://doi.org/10.1007/s40593-020-00228-w · DOI
  34. Rebecca Quintana, Chris Quintana, Cheryl Madeira, and James D. Slotta. 2016. Keeping Watch: Exploring Wearable Technology Designs for K-12 Teachers (CHI EA '16). Association for Computing Machinery, New York, NY, USA, 2272-2278. https://doi.org/10.1145/2851581.2892493 · DOI
  35. Mirko Raca and Pierre Dillenbourg. 2013. System for Assessing Classroom Atten- tion. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (Leuven, Belgium) (LAK '13). Association for Computing Machinery, New York, NY, USA, 265-269. https://doi.org/10.1145/2460296.2460351 · DOI
  36. Mike Sharples. 2013. Shared Orchestration Within and Beyond the Classroom. Computers & Education 69 (11 2013), 504-506. https://doi.org/10.1016/j.compedu. 2013.04.014 · DOI
  37. Mike Tissenbaum and Jim Slotta. 2019. Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. Int. J. Comput. Support. Collab. Learn. 14, 3 (Sept. 2019), 325-351. https://doi.org/10. 1007/s11412-019-09306-1 · DOI
  38. Kurt Vanlehn, Hugh Burkhardt, Salman Cheema, Seokmin Kang, Daniel Pead, Alan Schoenfeld, and Jon Wetzel. 2018. The Effect of Digital Versus Traditional Orchestration on Collaboration in Small Groups. 369-373. https://doi.org/10.1007/ 978-3-319-93846-2_69 · DOI
  39. Sharon Vaughn and Sylvia Linan-Thompson. 2003. What Is Special About Special Education for Students with Learning Disabilities? Journal of Special Education -J SPEC EDUC 37 (11 2003), 140-147. https://doi.org/10.1177/00224669030370030301 · DOI
  40. Erin Walker, Nikol Rummel, and Kenneth Koedinger. 2013. Adaptive Intelligent Support to Improve Peer Tutoring in Algebra. International Journal of Artificial Intelligence in Education 24 (10 2013). https://doi.org/10.1007/s40593-013-0001-9 · DOI
  41. Erin Walker, Nikol Rummel, and Kenneth R Koedinger. 2014. Adaptive Intelligent Support to Improve Peer Tutoring in Algebra. International Journal of Artificial Intelligence in Education 24, 1 (Jan. 2014), 33-61. https://doi.org/10.1007/s40593- 013-0001-9 · DOI
  42. Xu Wang, Meredith Thompson, Kexin Yang, Dan Roy, Kenneth Koedinger, Car- olyn Rose, and Justin Reich. 2021. Practice-Based Teacher Questioning Strategy Training with ELK: A Role-Playing Simulation for Eliciting Learner Knowl- edge. Proceedings of the ACM on Human-Computer Interaction 5 (04 2021), 1-27. https://doi.org/10.1145/3449125 · DOI
  43. Haoran Xie, Hui-Chun Chu, Gwo-Jen Hwang, and Chun-Chieh Wang. 2019. Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Computers & Education 140 (06 2019), 1-16. https://doi.org/10.1016/j.compedu.2019.103599 · DOI · Paper page
  44. Kexin Yang, Luettamae Lawrence, Vanessa Echeverria, Boyuan Guo, Nikol Rum- mel, and Vincent Aleven. 2021. Surveying Teachers' Preferences and Boundaries Regarding Human-AI Control in Dynamic Pairing of Students for Collaborative Learning. 260-274. https://doi.org/10.1007/978-3-030-86436-1_20 · DOI · Paper page

Cited by

Sorted by citations

Authors
Yang, K. B., Echeverria, V., Lu, Z., Mao, H., Holstein, K., Rummel, N., & Aleven, V.
Affiliations
  • Carnegie Mellon University (US)
  • Escuela Superior Politécnica del Litoral, Ecuador and Monash University, Australia
  • Ruhr University Bochum (DE)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-17
DOI
10.1145/3544548.3581398
OpenAlex
W4366549356
Cited by
47
References
6
Citations by year
2026: 19 · 2025: 15 · 2024: 13
FWCI
8.33
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • Monash University Research Portal (Monash University) (publishedVersion, other-oa)
Funders
The National Science Foundation (1822861)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
dynamical transition, intelligent tutoring systems, student pairing, classroom dynamics, teacher control, student control, personalized learning, smart classroom

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review