The DOI matches its Crossref record; 64 of the 65 references it cites were found in a registry or an index.
Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, a proliferation of studies has revealed the emergence of contrasting ethical themes relating to general AI and AIED, most of which are associated with the liability of data across settings, such as in higher education (Zawacki-Richter et al., 2019), K-12 (Holstein et al., 2019), schools (Luckin, 2017), and subjects (Hwang & Tu, 2021).”
From Ethical principles for artificial intelligence in education (2023) · cites this paper as Holstein et al., 2019
Citations
Cites
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 2,748
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Friedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966
(2012) · Choice Reviews Online · doi:10.5860/choice.49-5403 · Cited by 943
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930
Muller, M. J., & Kuhn, S. (1993) · Communications of the ACM · doi:10.1145/153571.255960 · Cited by 909
Intelligent tutoring systems and learning outcomes: A meta-analysis
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014) · Journal of Educational Psychology · doi:10.1037/a0037123 · Cited by 840
Cognitive Tutor: Applied research in mathematics education
RITTER, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. T. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194060 · Cited by 512
Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Holstein, K., McLaren, B. M., & Aleven, V. A. W. M. M. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.62.3 · Cited by 429
Self-citationStupid tutoring systems, intelligent humans
Baker, R. S. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0105-0 · Cited by 425
UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391
Design for classroom orchestration
Dillenbourg, P. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.04.013 · Cited by 386
Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2010) · Learning and Instruction · doi:10.1016/j.learninstruc.2010.07.004 · Cited by 386
Self-citationToward an Affect-Sensitive AutoTutor
D'Mello, S., Picard, R. W., & Graesser, A. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.79 · Cited by 386
Affect-aware tutors: recognising and responding to student affect
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009) · International Journal of Learning Technology · doi:10.1504/ijlt.2009.028804 · Cited by 371
Parallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367
Trischler, J., Pervan, S., Kelly, S. J., & Scott, D. R. (2017) · Journal of Service Research · doi:10.1177/1094670517714060 · Cited by 348
Help Helps, But Only So Much: Research on Help Seeking with Intelligent Tutoring Systems
Aleven, V. A. W. M. M., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0089-1 · Cited by 300
Self-citationAgency plus automation: Designing artificial intelligence into interactive systems
Heer, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807184115 · Cited by 279
Why Students Engage in “Gaming the System” Behavior in Interactive Learning Environments
Baker, R. S., Walonoski, J. A., Heffernan, N. T., Roll, I., Corbett, A. T., & Koedinger, K. R. (2008) · Cited by 258
Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235
Self-citationValue-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Informing Progress: Insights on Personalized Learning Implementation and Effects
Pane, J. F., Steiner, E. A., Baird, M., Hamilton, L. S., & Pane, J. (2017) · RAND Corporation eBooks · doi:10.7249/rr2042 · Cited by 205
Intelligent tutors as teachers' aides
Holstein, K., McLaren, B. M., & Aleven, V. A. W. M. M. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027451 · Cited by 194
Self-citationDecoding ClassDojo: psycho-policy, social-emotional learning and persuasive educational technologies
Williamson, B. (2017) · Learning Media and Technology · doi:10.1080/17439884.2017.1278020 · Cited by 192
Lee, M. K., & Baykal, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998230 · Cited by 182
Miller, J. K., Friedman, B., & Jancke, G. (2007) · doi:10.1145/1316624.1316668 · Cited by 181
Susan Bull, A., & Kay, J. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0090-8 · Cited by 180
Orchestrating technology enhanced learning: a literature review and a conceptual framework
Prieto, L. P., Dlab, M. H., Gutié, I., rrez, N. A., Abdulwahed, M. S., & Balid, W. (2011) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2011.045449 · Cited by 178
A fieldwork of the future with user enactments
Odom, W., Zimmerman, J., Davidoff, S., Forlizzi, J., Dey, A. K., & Lee, M. K. (2012) · Designing Interactive Systems Conference · doi:10.1145/2317956.2318008 · Cited by 176
Function allocation: algorithm, alchemy or apostasy?
Sheridan, T. B. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0285 · Cited by 175
Wheel-Spinning: Students Who Fail to Master a Skill
Beck, J. E., & Gong, Y. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_44 · Cited by 168
Rapidly exploring application design through speed dating
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007) · doi:10.5555/1771592.1771617 · Cited by 146
Adaptive Intelligent Support to Improve Peer Tutoring in Algebra
Walker, E., Rummel, N., & Koedinger, K. R. (2013) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-013-0001-9 · Cited by 142
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170377 · Cited by 129
Self-citationEliciting Values Reflections by Engaging Privacy Futures Using Design Workbooks
Wong, R. Y., Mulligan, D. K., Wyk, E. V., Pierce, J., & Chuang, J. C.-I. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134746 · Cited by 126
Mitchell, V., Ross, T., May, A. J., Sims, R. E., & Parker, C. (2015) · CoDesign · doi:10.1080/15710882.2015.1091894 · Cited by 104
Shared orchestration within and beyond the classroom
Sharples, M. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.04.014 · Cited by 97
Rapidly Exploring Application Design Through Speed Dating
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-74853-3_25 · Cited by 96
Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review
Rodríguez‐Triana, M. J., Prieto, L. P., Vozniuk, A., Boroujeni, M. S., Schwendimann, B. A., Holzer, A., & Gillet, D. (2017) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2017.084489 · Cited by 89
An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving
Alavi, H. S., & Dillenbourg, P. (2012) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2012.7 · Cited by 88
How Mastery Learning Works at Scale
RITTER, S., Yudelson, M. V., Fancsali, S. E., & Berman, S. R. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2876039 · Cited by 88
Speed Dating: Providing a Menu of Possible Futures
Zimmerman, J., & Forlizzi, J. (2017) · She ji · doi:10.1016/j.sheji.2017.08.003 · Cited by 76
Schofield, J. W., Eurich‐Fulcer, R., & Britt, C. L. (1994) · American Educational Research Journal · doi:10.3102/00028312031003579 · Cited by 72
Designing Future Employment Applications for Underserved Job Seekers
Dillahunt, T. R., Lam, J., Lu, A. J., & Wheeler, E. (2018) · Designing Interactive Systems Conference · doi:10.1145/3196709.3196770 · Cited by 70
Informing Teachers Live about Student Learning: Reporting in Assistment System
Feng, M., & Heffernan, N. T. (2005) · Cited by 66
Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems
Walonoski, J. A., & Heffernan, N. T. (2006) · Lecture notes in computer science · doi:10.1007/11774303_80 · Cited by 54
Using the Web-Based Interactive Software ALEKS to Enhance College Algebra
Hagerty, G., & Smith, S. (2005) · Mathematics and computer education · Cited by 51
Decision Tree Modeling of Wheel- Spinning and Productive Persistence in Skill Builders
Kai, S., Almeda, M. V., Baker, R. S., Heffernan, C., & Heffernan, N. T. (2018) · Journal of Educational Data Mining · doi:10.5281/zenodo.3344810 · Cited by 47
Intelligent teaching assistant systems
Yacef, K. (2003) · International Conference on Computers in Education · doi:10.1109/cie.2002.1185885 · Cited by 42
An, P., Bakker, S., Ordanovski, S., Taconis, R., & Eggen, B. H. (2018) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/3173225.3173243 · Cited by 37
Echeverría, V., Martínez‐Maldonado, R., Power, T., Hayes, C., & Shum, S. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93846-2_14 · Cited by 35
Dillenbourg, P., Prieto, L. P., & Olsen, J. K. (2018) · Routledge eBooks · doi:10.4324/9781315617572-18 · Cited by 28
Supporting orchestration of blended CSCL scenarios in distributed learning environments
Prieto, L. P. (2012) · UVaDOC UVaDOC University of Valladolid Documentary Repository (University of Valladolid) · doi:10.35376/10324/1794 · Cited by 23
Schofield, J. W. (1997) · Social Science Computer Review · doi:10.1177/089443939701500104 · Cited by 21
Informing the Design of Teacher Awareness Tools through Causal Alignment Analysis.
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · International Society of the Learning Sciences, Inc. [ISLS]. eBooks · doi:10.22318/cscl2018.104 · Cited by 20
Self-citationIncreasing Parent Engagement in Student Learning Using an Intelligent Tutoring System
Broderick, Z. R., O’Connor, C., Mulcahy, C., Heffernan, N. T., & Heffernan, C. (2011) · Digital WPI · Cited by 14
A Preliminary Evaluation of the Usability of an AI-Infused Orchestration System
Wetzel, J., Burkhardt, H., Cheema, S. A., Kang, S., Pead, D., Schoenfeld, A. H., & VanLehn, K. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93846-2_71 · Cited by 12
Long, Y., Aman, Z., & Aleven, V. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_23 · Cited by 11
Self-citationvan Leeuwen, A., Rummel, N., Aleven, V. A. W. M. M., Gal, K., Holstein, K., Campen, C. A. N. K., McLaren, B. M., Molenaar, I., Prusak, N., Schwarz, B. B., et al. (2018) · International Conference of Learning Sciences · doi:10.22318/cscl2018.1227 · Cited by 9
Self-citationOrchestrating Combined Collaborative and Individual Learning in the Classroom
Olsen, J. K. (2017) · KiltHub Repository · doi:10.1184/r1/6720983 · Cited by 9
Towards Integrating Human and Automated Tutoring Systems.
RITTER, S., Yudelson, M. V., Fancsali, S. E., & Berman, S. (2016) · Educational Data Mining · Cited by 7
Calculating children in the dataveillance school: personal and learning analytics
Williamson, B. (2016) · doi:10.4324/9781315611402-10 · Cited by 7
Luckin, R., & Clark, W. (2011) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.20368/1971-8829/550 · Cited by 3
Research works printed but not linked: 2
- Mulligan, D.K., King, J.: Bridging the gap between privacy and design. U. Pa. J. Const. L. 14, 989 (2011)
- Watters, A.: The Monsters of Education Technology. Smashwords Edition (2014)
Cited by
Sorted by citations
Ethics of AI in Education: Towards a Community-Wide Framework
Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., et al. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00239-1 · Cited by 1,437
“In addition, there remains limited research into what teachers and students actually want from AIED systems such as requirements around student agency and privacy about which teachers and students might not agree (Holstein et al. 2019).”
BackgroundSelf-citationEthical principles for artificial intelligence in education
Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11316-w · Cited by 1,228
“Indeed, a proliferation of studies has revealed the emergence of contrasting ethical themes relating to general AI and AIED, most of which are associated with the liability of data across settings, such as in higher education (Zawacki-Richter et al., 2019), K-12 (Holstein et al., 2019), schools (Luckin, 2017 ), and su…”
BackgroundAI literacy in K-12: a systematic literature review
Casal-Otero, L., Catala, A., Fernández-Morante, C., Taboada, M., Cebreiro, B., & Barro, S. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00418-7 · Cited by 724
“, 2020), their opinions on the tools that support the teaching–learning process in AI (Holstein et al., 2019) and their teacher training in technologies (Cheung et al, 2018; Jaskie et al.”
BackgroundInfluentialMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436
Leading teachers' perspective on teacher-AI collaboration in education
Kim, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12109-5 · Cited by 376
Metadata
- Authors
- Holstein, K., McLaren, B. M., & Aleven, V.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Holstein, Kenneth
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2019
- Pages
- 157-171
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2952696298
- Cited by
- 189
- References
- 65
- Citations by year
- 2026: 46 · 2025: 45 · 2024: 29 · 2023: 21 · 2022: 21 · 2021: 14 · 2020: 12 · 2019: 1
- FWCI
- 4.59
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- human-AI collaboration, artificial intelligence in education, K-12 education, teacher needs, student needs, AI tutors, human tutoring, student motivation, emotional support, human-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
