The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Open accessAbstract
Involving stakeholders throughout the creation of new educational technologies can help ensure their usefulness and usability in real-world contexts. However, given the complexity of learning analytics (LA) systems, it can be challenging to meaningfully involve non-technical stakeholders throughout their design and development. This article reports on the iterative co-design, development, and classroom evaluation of Konscia, a wearable, real-time awareness tool for teachers working in AI-enhanced K-12 classrooms. In the process, we argue that the co-design of LA systems requires new kinds of prototyping methods. We introduce one of our own prototyping methods, REs, to address unique challenges of co-prototyping LA tools. This work presents the first end-to-end demonstration of how non-technical stakeholders can participate throughout the whole design process for a complex LA system—from early generative phases to the selection and tuning of analytics to evaluation in real-world contexts. We conclude by providing methodological recommendations for future LA co-design efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chapter 12: What does the Anthropocene hold for citizenship?
(2019) · edoc (University of Basel) · doi:10.1007/978-3 · Cited by 11,533
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 2,748
An introduction to human factors engineering
Wickens, C. D. (1997) · Cited by 1,711
Participatory Design: Principles and Practices
(2017) · CRC Press eBooks · doi:10.1201/9780203744338 · Cited by 1,500
Buchenau, M., & Suri, J. F. (2000) · Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques · doi:10.1145/347642.347802 · Cited by 1,237
(2012) · Choice Reviews Online · doi:10.5860/choice.49-5403 · Cited by 960
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Self-citationManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
Research Methods for Human–Computer Interaction
Bias, R. G. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21187 · Cited by 720
Heffernan, N. T., & Heffernan, C. L. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0024-x · Cited by 616
A review of recent advances in learner and skill modeling in intelligent learning environments
Desmarais, M. C., & Baker, R. S. J. D. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9106-8 · Cited by 449
Stupid tutoring systems, intelligent humans
Baker, R. S. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0105-0 · Cited by 425
UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391
Understanding contexts by being there: case studies in bodystorming
Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003) · Personal and Ubiquitous Computing · doi:10.1007/s00779-003-0238-7 · Cited by 344
A Data Repository for the EDM Community: The PSLC DataShop
(2010) · Handbook of Educational Data Mining · doi:10.1201/b10274-10 · Cited by 322
Participatory design: principles and practices
Schuler, D., Namioka, A., & Suchman, L. A. (1993) · CERN Document Server (European Organization for Nuclear Research) · Cited by 307
DESIGNING FORMATIVE ASSESSMENT SOFTWARE WITH TEACHERS: AN ANALYSIS OF THE CO-DESIGN PROCESS
PENUEL, W. R., ROSCHELLE, J., & SHECHTMAN, N. (2007) · Research and Practice in Technology Enhanced Learning · doi:10.1142/s1793206807000300 · Cited by 300
Help Helps, But Only So Much: Research on Help Seeking with Intelligent Tutoring Systems
Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0089-1 · Cited by 255
Self-citationStudent Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235
Self-citationChildren's active free play in local neighborhoods: a behavioral mapping study
Veitch, J., Salmon, J., & Ball, K. (2007) · Health Education Research · doi:10.1093/her/cym074 · Cited by 227
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Lee, M. K., & Baykal, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998230 · Cited by 182
A fieldwork of the future with user enactments
Odom, W., Zimmerman, J., Davidoff, S., Forlizzi, J., Dey, A. K., & Lee, M. K. (2012) · Proceedings of the Designing Interactive Systems Conference · doi:10.1145/2317956.2318008 · Cited by 175
Holstein, K., McLaren, B. M., & Aleven, V. (2017) · Proceedings of the Seventh International Learning Analytics & Knowledge Conference · doi:10.1145/3027385.3027451 · Cited by 173
Self-citationBull, S., & Kay, J. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0090-8 · Cited by 164
Wheel-Spinning: Students Who Fail to Master a Skill
Beck, J. E., & Gong, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39112-5_44 · Cited by 161
Directions in hybrid intelligence: complementing AI systems with human intelligence
Kamar, E. S. (2016) · International Joint Conference on Artificial Intelligence · Cited by 154
Rapidly exploring application design through speed dating
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007) · doi:10.5555/1771592.1771617 · Cited by 146
Comicboarding: using comics as proxies for participatory design with children
Moraveji, N., Li, J., Ding, J., O'Kelley, P., & Woolf, S. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240832 · Cited by 135
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170377 · Cited by 129
Self-citationToward human-centered algorithm design
Baumer, E. P. (2017) · Big Data & Society · doi:10.1177/2053951717718854 · Cited by 125
An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment
Maldonado, R. M., Kay, J., Yacef, K., & Schwendimann, B. A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_62 · Cited by 112
Martínez‐Maldonado, R., Clayphan, A. J., Yacef, K., & Kay, J. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2014.2365027 · Cited by 106
Rapidly Exploring Application Design Through Speed Dating
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-74853-3_25 · Cited by 101
Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review
Rodríguez‐Triana, M. J., Prieto, L. P., Vozniuk, A., Boroujeni, M. S., Schwendimann, B. A., Holzer, A., & Gillet, D. (2017) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2017.084489 · Cited by 91
User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback
Tohidi, M., Buxton, W. A. S., Baecker, R. M., & Sellen, A. J. (2006) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1182475.1182487 · Cited by 85
Co-creation strategies for learning analytics
Dollinger, M., & Lodge, J. M. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170372 · Cited by 81
Feedback in questioning and marking : the science teacher's role in formative assessment
Black, P., & Harrison, C. A. (2001) · Research Portal (King's College London) · Cited by 73
Martínez‐Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. J. (2016) · Journal of Learning Analytics · doi:10.18608/jla.2015.23.3 · Cited by 73
Schofield, J. W., Eurich‐Fulcer, R., & Britt, C. L. (1994) · American Educational Research Journal · doi:10.3102/00028312031003579 · Cited by 72
Towards Understanding How to Assess Help-Seeking Behavior Across Cultures
Ogan, A., Walker, E., Baker, R. S., Rodrigo, M. M. T., Soriano, J. C. A., & Jiménez-Castro, M. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0034-8 · Cited by 68
Decision Tree Modeling of Wheel- Spinning and Productive Persistence in Skill Builders
Kai, S., Almeda, M. V., Baker, R. S., Heffernan, C., & Heffernan, N. T. (2018) · Journal of Educational Data Mining · doi:10.5281/zenodo.3344810 · Cited by 55
Co-designing learning analytics tools with learners
Alvarez, C. P., Martínez‐Maldonado, R., & Anderson, T. D. (2018) · Routledge eBooks · doi:10.4324/9781351113038-7 · Cited by 55
Function allocation: a perspective from studies of work practice
Wright, P., Dearden, A., & Fields, B. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0292 · Cited by 49
Design and evaluation of teacher assistance tools for exploratory learning environments
Mavrikis, M., Gutiérrez-Santos, S., & Poulovassilis, A. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883909 · Cited by 46
Khachatryan, G. A., Romashov, A. V., Khachatryan, A. R., Gaudino, S., Khachatryan, J. M., Guarian, K. R., & Yufa, N. V. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0019-7 · Cited by 44
Penuel, W. R., & Yarnall, L. (2005) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 43
Collaboration in cognitive tutor use in latin America
Ogan, A., Walker, E., Baker, R. S., Rebolledo‐Mendez, G., Jiménez-Castro, M., Laurentino, T., & de Carvalho, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208597 · Cited by 43
Intelligent teaching assistant systems
Yacef, K. (2003) · International Conference on Computers in Education · doi:10.1109/cie.2002.1185885 · Cited by 42
SPACLE: investigating learning across virtual and physical spaces using spatial replays
Holstein, K., McLaren, B. M., & Aleven, V. A. W. M. M. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027450 · Cited by 37
Self-citationDesigning for Democracy in Education: Participatory Design and the Learning Sciences.
DiSalvo, B. J., & DiSalvo, C. F. (2014) · Proceedings. · Cited by 26
Käser, T., Klingler, S., & Groß, M. H. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883961 · Cited by 23
Aguilar, S. J. (2017) · Journal of Research on Technology in Education · doi:10.1080/15391523.2017.1401498 · Cited by 20
Opening Up an Intelligent Tutoring System Development Environment for Extensible Student Modeling
Holstein, K., Yu, Z., Sewall, J., Popescu, O., McLaren, B. M., & Aleven, V. A. W. M. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93843-1_13 · Cited by 20
Self-citationInforming the Design of Teacher Awareness Tools through Causal Alignment Analysis.
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · International Society of the Learning Sciences, Inc. [ISLS]. eBooks · doi:10.22318/cscl2018.104 · Cited by 20
Self-citationMavrikis, M., Gutiérrez-Santos, S., Geraniou, E., Hoyles, C., Magoulas, G. D., Noss, R., & Poulovassilis, A. (2015) · Routledge eBooks · doi:10.4324/9780203075227.ch7 · Cited by 10
Orchestrating Combined Collaborative and Individual Learning in the Classroom
Olsen, J. K. (2017) · KiltHub Repository · doi:10.1184/r1/6720983 · Cited by 9
Cerri, S. A., Clancey, W. J., Papadourakis, G. M., & Panourgia, K. (2012) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.1007/978-3-642-30950-2 · Cited by 8
Towards Integrating Human and Automated Tutoring Systems.
RITTER, S., Yudelson, M. V., Fancsali, S. E., & Berman, S. (2016) · Educational Data Mining · Cited by 7
Rau, M. A. (2015) · Cited by 3
Cited by
Chiu, T. K. F., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100118 · Cited by 1,181
Celik, I. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107468 · Cited by 1,012
The promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984
Jeon, J., & Lee, S. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11834-1 · Cited by 696
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
Metadata
- Authors
- Holstein, K., McLaren, B. M., & Aleven, V.
- Venue
- Journal of Learning Analytics
- Published
- 2019-07-22
- Volume
- 6
- Issue
- 2
- ISSN
- 1929-7750 (ISSN-L)
- OpenAlex
- W2966032592
- Cited by
- 390
- References
- 63
- FWCI
- 23.42
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.