Intelligent tutors as teachers' aides: exploring teacher needs for real-time analytics in blended classrooms
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Intelligent tutoring systems (ITSs) are commonly designed to enhance student learning. However, they are not typically designed to meet the needs of teachers who use them in their classrooms. ITSs generate a wealth of analytics about student learning and behavior, opening a rich design space for real-time teacher support tools such as dashboards. Whereas real-time dashboards for teachers have become popular with many learning technologies, we are not aware of projects that have designed dashboards for ITSs based on a broad investigation of teachers' needs. We conducted design interviews with ten middle school math teachers to explore their needs for on-the-spot support during blended class sessions, as a first step in a user-centered design process of a real-time dashboard. Based on multi-methods analyses of this interview data, we identify several opportunities for ITSs to better support teachers' needs, noting that the analytics commonly generated by existing teacher support tools do not strongly align with the analytics teachers expect to be most useful. We highlight key tensions and tradeoffs in the design of such real-time supports for teachers, as revealed by "Speed Dating" possible futures with teachers. This paper has implications for our ongoing co-design of a real-time dashboard for ITSs, as well as broader implications for the design of ITSs that can effectively collaborate with teachers in classroom settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324 · cited 1×
“These student models may include moment-by-moment estimates of student knowledge (e.g., [38]), whether a student seems “stuck” on a given skill (e.g., [19, 32]) or engaged in their current activity (e.g., [34]), whether a student is exhibiting particular misconceptions (e.g., [3]), and so on.”
BackgroundStupid tutoring systems, intelligent humans
Baker, R. S. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0105-0 · Cited by 425 · cited 2×
“Others have proposed similar directions, from the lens of optimizing student learning: ITSs might better facilitate student learning if these systems could both help and leverage human teachers, when teachers are present in the classroom [4, 9, 11, 28].”
BackgroundYang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 1×
“…effective patterns of behavior, on the one hand, and the need to privilege teachers’ autonomy and rich prior knowledge, on the other (paralleling recent findings in other domains where intelligent systems are developed to support human experts’ decision-making, such as healthcare [42]).”
BackgroundBarriers to ITS Adoption: A Systematic Mapping Study
Nye, B. D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-07221-0_74 · Cited by 20
Research works printed but not linked: 38
- 10.1007/s10972-016-9459-2 · DOI
- Martinez-Maldonado, R., Dimitriadis, Y., Kay, J., Yacef, K., & Edbauer, M. T. (2013). MTClassroom and MTDashboard: supporting analysis of teacher attention in an orchestrated multi-tabletop classroom. In Proc. CSCL2013, 119--128.
- 10.1007/978-3-319-45153-4_26 · DOI
- 10.5555/1771592.1771617 · doi:10.5555/1771592.1771617 · Paper page
- Hanington, B., & Martin, B. (2012). Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. Rockport Publishers.
- 10.5555/838238.839178 · doi:10.5555/838238.839178
- Feng, M., & Heffernan, N. T. (2006). Informing teachers live about student learning: Reporting in the Assistment system. Technology Instruction Cognition and Learning, 3, 1--8.
- 10.5555/876869.878017 · doi:10.5555/876869.878017
- 10.1007/978-3-642-77681-6_11 · DOI
- 10.1145/122652.122660 · doi:10.1145/122652.122660
- Tissenbaum, M. & Matuk, C. (2016). Real-time visualization of student activities to support classroom orchestration. In ICLS 2016, 1120--1127.
- 10.5555/1457554 · doi:10.5555/1457554
- Pinkwart, N. (2016). Another 25 years of AIED? Challenges and opportunities for intelligent educational technologies of the future. IJAIED, 26(2), 771--783.
- 10.5555/523184 · doi:10.5555/523184
- Evenson, S. (2006). Directed storytelling: Interpreting experience for design. Design Studies: Theory and research in graphic design, 231--240.
- Corbin, J., & Strauss, A. (2014). Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
- 10.1007/978-3-642-39112-5_44 · DOI · Paper page
- 10.5555/1562524.1562600 · doi:10.5555/1562524.1562600
- 10.5555/1898078.1898079 · doi:10.5555/1898078.1898079
- 10.1145/2090116.2090129 · DOI · Paper page
- 10.1007/978-3-319-24258-3_20 · DOI
- Rodriguez Triana M. J. Prieto L. P. Vozniuk A. Shirvani Boroujeni M. Schwendimann B. A. Holzer A. C. & Gillet D. (in press). Monitoring Awareness and Reflection in Blended Technology Enhanced Learning: a Systematic Review. IJTEL.
- 10.5555/648031.746165 · doi:10.5555/648031.746165
- 10.1007/978-1-4419-5716-0_26 · DOI · Paper page
- 10.1145/2883851.2883927 · DOI · Paper page
- 10.1007/978-3-642-13437-1_85 · doi:10.1007/978-3-642-13437-1_85
- 10.1145/2723576.2723607 · DOI · Paper page
- Matuk, C., Gerard, L., Lim-Breitbart, J., & Linn, M. C. (2016). Teachers' reflections on the uses of real-time data in their instruction. Poster session presented at AERA 2016, Washington, DC, USA.
- 10.1145/2876034.2876039 · DOI · Paper page
- 10.1145/2883851.2883961 · DOI
- 10.1145/2883851.2883967 · DOI
- Baker, R.S.J.d, Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., Salvi, A., Aleven, V., Kusbit, G., Ocumpaugh, J., & Rossi, L. (2012). Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra. In EDM 2012, 126--133.
- 10.5555/1050231.1050284 · doi:10.5555/1050231.1050284
- 10.3758/BF03194060 · DOI · Paper page
- 10.1007/978-1-4419-5546-3_7 · DOI
- 10.1007/978-0-387-09673-5_4 · DOI
- 10.4324/9780203832714 · DOI · Paper page
- 10.1145/3027385.3027450 · DOI · Paper page
Cited by
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
“rXiv:1812.05239v2 [cs.HC] 7 Jan 2019 being prototyped with practitioners, their initial design often appears to be driven more by the availability of algorithmic methods than by real-world needs (cf. [52, 111]). If such tools are to have a positive and meaningful impact on industry practice, however, it is crucial tha…”
BackgroundSelf-citationThe promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Bernacki, M. L., Greene, J. A., & Crompton, H. (2020) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2019.101827 · Cited by 484
Chounta, I.-A., Bardone, E., Raudsep, A., & Pedaste, M. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00243-5 · Cited by 441
A review on artificial intelligence in education
Huang, J., Saleh, S., & Liu, Y. (2021) · Academic Journal of Interdisciplinary Studies · doi:10.36941/ajis-2021-0077 · Cited by 439
Nazari, N., Shabbir, M. S., & Setiawan, R. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07014 · Cited by 405
Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.62.3 · Cited by 390
Self-citationLeading teachers' perspective on teacher-AI collaboration in education
Kim, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12109-5 · Cited by 376
Jin, S.-H., Im, K., Yoo, M., Roll, I., & Seo, K. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00406-5 · Cited by 332
Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235
Self-citationHolstein, K., McLaren, B. M., & Aleven, V. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-23204-7_14 · Cited by 189
Self-citationA Conceptual Framework for Human–AI Hybrid Adaptivity in Education
Holstein, K., Aleven, V., & Rummel, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52237-7_20 · Cited by 168
Self-citationDesigning for human–AI complementarity in K‐12 education
Holstein, K., & Aleven, V. (2022) · AI Magazine · doi:10.1002/aaai.12058 · Cited by 156
Self-citationKochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00267-x · Cited by 112
Driving data storytelling from learning design
Echeverria, V., Martinez-Maldonado, R., Granda, R., Chiluiza, K., Conati, C., & Buckingham Shum, S. (2018) · Proceedings of the 8th International Conference on Learning Analytics and Knowledge · doi:10.1145/3170358.3170380 · Cited by 80
Educator perspectives on learning analytics in classroom practice
Shibani, A., Knight, S., & Buckingham Shum, S. (2020) · The Internet and Higher Education · doi:10.1016/j.iheduc.2020.100730 · Cited by 86
Echeverria, V., Martinez-Maldonado, R., Buckingham Shum, S., Chiluiza, K., Granda, R., & Conati, C. (2018) · Journal of Learning Analytics · doi:10.18608/jla.2018.53.6 · Cited by 70
“Furthermore, the choice of which data is to be visualized does not commonly correspond with what students and educators really want or need, in part, because they are not regularly consulted as part of the design process (Holstein et al., 2017).”
BackgroundThe TA Framework: Designing Real-time Teaching Augmentation for K-12 Classrooms
An, P., Holstein, K., d'Anjou, B., Eggen, B., & Bakker, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376277 · Cited by 69
“Prior design research suggests that teachers sometimes find the kinds of information commonly presented in learning analytics dashboards to be too lowlevel to be useful [54, 107].”
BackgroundInfluentialSelf-citationThe equalizing effect of teacher dashboards on feedback in K-12 classrooms
Knoop-van Campen, C. A. N., Wise, A., & Molenaar, I. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1931346 · Cited by 58
Kawakami, A., Coston, A., Zhu, H., Heidari, H., & Holstein, K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642849 · Cited by 54
Self-citationMy teammate is AI: understanding students’ perceptions of student-AI collaboration in drawing tasks
Kim, J., & Cho, Y. H. (2025) · Asia Pacific Journal of Education · doi:10.1080/02188791.2023.2286206 · Cited by 50
“However, when human agency is too high, humans may experience algorithm aversion, the tendency to depend on human judgements over algorithmic suggestions yet the former is suboptimal (Holstein et al., 2017).”
BackgroundYang, K. B., Echeverria, V., Lu, Z., Mao, H., Holstein, K., Rummel, N., & Aleven, V. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581398 · Cited by 47
“Educational technology works best when aligned with teachers’ values, instructional goals, and teaching practices [23, 28].”
BackgroundSelf-citationDesigning Appropriate Learning Technologies for School vs Home Settings in Tanzanian Rural Villages
Uchidiuno, J., Yarzebinski, E., Madaio, M., Maheshwari, N., Koedinger, K., & Ogan, A. (2018) · Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies · doi:10.1145/3209811.3209881 · Cited by 34
Supporting teachers' intervention in students' virtual collaboration using a network based model
Herder, T., Swiecki, Z., Fougt, S. S., Tamborg, A. L., Allsopp, B. B., Shaffer, D. W., & Misfeldt, M. (2018) · Proceedings of the 8th International Conference on Learning Analytics and Knowledge · doi:10.1145/3170358.3170394 · Cited by 26
Kawakami, A., Coston, A., Heidari, H., Holstein, K., & Zhu, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686989 · Cited by 24
Self-citationYang, K. B., Lawrence, L., Echeverria, V., Guo, B., Rummel, N., & Aleven, V. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-86436-1_20 · Cited by 19
Self-citationChang, M. A., Tissenbaum, M., Philip, T. M., & D’Mello, S. K. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100364 · Cited by 12
Into the Unknown: AI Adoption and Job Satisfaction of Christian Faculty
Manicone, H. A., B, R., FelixOtuorimuo, I. E., Graf, L. M., Joseph, R. A., Moore, E. D., & Wilson, J. E. (2026) · Journal for the Scholarship of Teaching & Learning · doi:10.70623/ohtr6058
Metadata
- Authors
- Holstein, K., McLaren, B. M., & Aleven, V.
- Venue
- Proceedings of the Seventh International Learning Analytics & Knowledge Conference
- Publisher
- ACM
- Published
- 2017-03-13
- Pages
- 257-266
- OpenAlex
- W2592229992
- Cited by
- 173
- References
- 4
- FWCI
- 7.24
- Type
- conference paper
- Language
- en
- Open access
- bronze
- Funders
- National Science Foundation (1530726)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.