CoChat
Sign Up
Cited by 20Digital Education and SocietyEthics and Social Impacts of AIArtificial Intelligence in Education

Learning in the Panopticon: Examining the Potential Impacts of AI Monitoring on Students

Open access
Han, Bingyi, Buchanan, George, Mckay, Dana·2022·Proceedings of the 34th Australian Conference on Human-Computer Interaction
DOI 10.1145/3572921.3572937
Open this paper in CoChat — free accountOpen PDF

Abstract

In a panopticon, people are intrusively monitored across all areas of their lives. AI monitoring has been ever more widely adopted in education, with increasingly intrusive monitoring of students. These changes potentially create ethical harms, but current ethical discussions predominantly focus on legal and governance issues. The concerns of the majority of users—namely students—are neglected. Overlooking students’ concerns further increases their vulnerability. We use a student-centred and speculative approach through the Story Completion Method (SCM) to explore how students would potentially respond to intrusive AI monitoring in a higher education setting. Our study included 71 participants who elaborated on the story stems we provided to them. Through a blending of thematic analysis coding and the techniques of developing grounded theory, we reveal that the common responses of students to extensive AI monitoring included impacts on personal psychology, changed behaviour, and cognition. There are likely major disruptions to personal autonomy, identity and educational relationships. If we are to avoid a future ‘big brother’ classroom, further investigations using HCI methods are critical to understanding how to protect students in AI-dominated learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3572921.3572937

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Romero, C. and S. Ventura, Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2013. 3(1): p. 12-27.
  2. Marcu, G. and A.N. Spiller. Collaborative Aspects of Collecting and Reflecting on Behavioral Data. in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020.
  3. Siemens, G. and P. Long, Penetrating the fog: Analytics in learning and education. EDUCAUSE review, 2011. 46(5): p. 30.
  4. Mangaroska, K. and M. Giannakos, Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 2018. 12(4): p. 516-534.
  5. Arnold, K.E. and N. Sclater. Student perceptions of their privacy in learning ana- lytics applications. in Proceedings of the seventh international learning analytics & knowledge conference. 2017.
  6. Schiff, D., et al. What's next for ai ethics, policy, and governance? a global overview. in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2020.
  7. Wise, A., Y. Zhao, and S. Hausknecht, Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 2014. 1(2): p. 48-71.
  8. Kumar, P.C., et al. Privacy and security considerations for digital technology use in elementary schools. in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.
  9. Dewan, M., M. Murshed, and F. Lin, Engagement detection in online learning: a review. Smart Learning Environments, 2019. 6(1): p. 1-20.
  10. Ez-Zaouia, M., A. Tabard, and E. Lavoué, EMODASH: A dashboard supporting retrospective awareness of emotions in online learning. International Journal of Human-Computer Studies, 2020. 139: p. 102411.
  11. Dunne, A. and F. Raby, Speculative everything: design, fiction, and social dream- ing. 2013: MIT press.
  12. Wong, R.Y. and V. Khovanskaya, Speculative design in HCI: from corporate imaginations to critical orientations, in New Directions in Third Wave Human- Computer Interaction: Volume 2-Methodologies. 2018, Springer. p. 175-202.
  13. Blythe, M., et al. Imaginary design workbooks: Constructive criticism and practi- cal provocation. in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018.
  14. Troiano, G.M., J. Tiab, and Y.-K. Lim. SCI-FI: shape-changing interfaces, future interactions. in Proceedings of the 9th Nordic Conference on Human-Computer Interaction. 2016.
  15. Lawson, S., et al. Problematising upstream technology through speculative design: the case of quantified cats and dogs. in Proceedings of the 33rd annual ACM conference on human factors in computing systems. 2015.
  16. Wood, M., G. Wood, and M. Balaam. " They're Just Tixel Pits, Man" Disputing the'Reality'of Virtual Reality Pornography through the Story Completion Method. in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017.
  17. Holstein, K., B.M. McLaren, and V. Aleven. SPACLE: Investigating learning across virtual and physical spaces using spatial replays. in Proceedings of the Seventh International Learning Analytics & Knowledge Conference. 2017.
  18. Holstein, K., B.M. McLaren, and V. Aleven. Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms. 2018. Cham: Springer International Publishing.
  19. Ez-Zaouia, M. and E. Lavoué. EMODA: A tutor oriented multimodal and contex- tual emotional dashboard. in Proceedings of the seventh international learning analytics & knowledge conference. 2017.
  20. Lim, L., et al. Exploring students' sensemaking of learning analytics dashboards: Does frame of reference make a difference? in Proceedings of the 9th international conference on learning analytics & knowledge. 2019.
  21. Noroozi, O., et al., Multimodal data indicators for capturing cognitive, motiva- tional, and emotional learning processes: A systematic literature review. Educa- tion and Information Technologies, 2020. 25(6): p. 5499-5547.
  22. Park, Y. and I.-H. Jo, Development of the learning analytics dashboard to support students' learning performance. Journal of Universal Computer Science, 2015. 21(1): p. 110.
  23. Li, X., et al., Massive Open Online Proctor: Protecting the Credibility of MOOCs certificates, in Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. 2015, Association for Computing Ma- chinery: Vancouver, BC, Canada. p. 1129-1137.
  24. Sung, G., T. Feng, and B. Schneider, Learners Learn More and Instructors Track Better with Real-time Gaze Sharing. Proc. ACM Hum.-Comput. Interact., 2021. 5(CSCW1): p. Article 134.
  25. Lundie, D., Authority, autonomy and automation: The irreducibility of pedagogy to information transactions. Studies in philosophy and education, 2016. 35(3): p. 279-291.
  26. Drachsler, H., et al. Ethical and privacy issues in the application of learning ana- lytics. in Proceedings of the fifth international conference on learning analytics and knowledge. 2015.
  27. Lu, A.J., et al., Coding Bias in the Use of Behavior Management Technologies: Uncovering Socio-technical Consequences of Data-driven Surveillance in Class- rooms, in Designing Interactive Systems Conference 2021. 2021, Association for Computing Machinery: Virtual Event, USA. p. 508-522.
  28. Durán, J.M. and K.R. Jongsma, Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. Journal of Medical Ethics, 2021. 47(5): p. 329-335.
  29. Prinsloo, P. and S. Slade, Student vulnerability, agency, and learning analytics: An exploration. Journal of Learning Analytics, 2016. 3(1): p. 159-182.
  30. Sclater, N., Developing a Code of Practice for Learning Analytics. Journal of Learning Analytics, 2016. 3(1): p. 16-42.
  31. Lu, A.J., et al., Data Work in Education: Enacting and Negotiating Care and Control in Teachers' Use of Data-Driven Classroom Surveillance Technology. Proceedings of the ACM on Human-Computer Interaction, 2021. 5(CSCW2): p. 1-26.
  32. Whitelock-Wainwright, A., et al., The Student Expectations of Learning Analytics Questionnaire. Journal of Computer Assisted Learning, 2019. 35(5): p. 633-666.
  33. Jones, K.M., et al., "We're being tracked at all times": Student perspectives of their privacy in relation to learning analytics in higher education. Journal of the Association for Information Science and Technology, 2020. 71(9): p. 1044-1059.
  34. Tsai, Y.-S., A. Whitelock-Wainwright, and D. Gašević, The privacy paradox and its implications for learning analytics, in Proceedings of the Tenth International Con- ference on Learning Analytics & Knowledge. 2020, Association for Computing Machinery: Frankfurt, Germany. p. 230-239.
  35. Troiano, G.M., et al. "Are We Now Post-COVID?": Exploring Post-COVID Futures Through a Gamified Story Completion Method. in Designing Interactive Systems Conference 2021. 2021.
  36. Troiano, G.M., M. Wood, and C. Harteveld. " And This, Kids, Is How I Met Your Mother": Consumerist, Mundane, and Uncanny Futures with Sex Robots. in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020.
  37. Clarke, V., et al., Editorial introduction to the special issue: using story completion methods in qualitative research. 2019, Taylor & Francis. p. 1-20.
  38. Ma, Y., Relational privacy: Where the East and the West could meet. Proceedings of the Association for Information Science and Technology, 2019. 56(1): p. 196- 205.
  39. Prinsloo, P. and S. Slade, Student Vulnerability, Agency and Learning Analytics: An Exploration. Journal of Learning Analytics, 2016. 3(1): p. 159-182.
  40. Breckler, S.J., Empirical validation of affect, behavior, and cognition as distinct components of attitude. Journal of personality and social psychology, 1984. 47(6): p. 1191.
  41. Oulasvirta, A., et al., Long-term effects of ubiquitous surveillance in the home, in Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 2012, Association for Computing Machinery: Pittsburgh, Pennsylvania. p. 41-50.
  42. Foucault, M., Security, territory, population: lectures at the Collège de France, 1977-78. 2007: Springer.
  43. Li, W., et al., Disparities in students' propensity to consent to learning analytics. International Journal of Artificial Intelligence in Education, 2021: p. 1-45.
  44. Ball, S.J., The teacher's soul and the terrors of performativity. Journal of education policy, 2003. 18(2): p. 215-228.
  45. Chen, B. and H. Zhu, Towards Value-Sensitive Learning Analytics Design, in Pro- ceedings of the 9th International Conference on Learning Analytics & Knowledge. 2019, Association for Computing Machinery: Tempe, AZ, USA. p. 343-352.
  46. Alkhatib, S., et al. Privacy and the Internet of Things (IoT) Monitoring Solutions for Older Adults: A. in Connecting the System to Enhance the Practitioner and Consumer Experience in Healthcare: Selected Papers from the 26th Australian National Health Informatics Conference (HIC 2018). 2018. IOS Press.
  47. Ur, B., J. Jung, and S. Schechter. Intruders versus intrusiveness: teens' and parents' perspectives on home-entryway surveillance. in Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2014.
  48. Cranor, L.F., et al. {Parents'} and {Teens'} Perspectives on Privacy In a {Technology- Filled} World. in 10th Symposium On Usable Privacy and Security (SOUPS 2014). 2014.
  49. Ghosh, A.K., et al. A matter of control or safety? Examining parental use of technical monitoring apps on teens' mobile devices. in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018.
  50. Levy, K. and B. Schneier, Privacy threats in intimate relationships. Journal of Cybersecurity, 2020. 6(1): p. tyaa006.
  51. Erkin, Z., et al., Privacy-Preserving Face Recognition. 2009, Springer Berlin Hei- delberg. p. 235-253.
  52. Stahl, B.C. and D. Wright, Ethics and privacy in AI and big data: Implementing responsible research and innovation. IEEE Security & Privacy, 2018. 16(3): p. 26-33.
  53. Baek, C. and S.J. Aguilar, Past, present, and future directions of learning analytics research for students with disabilities. Journal of Research on Technology in Education, 2022: p. 1-16.
  54. Dawson, S., et al. Rethinking learning analytics adoption through complexity leadership theory. in Proceedings of the 8th international conference on learning analytics and knowledge. 2018.
  55. Kelly, R.M., et al. "It's About Missing Much More Than the People": How Students use Digital Technologies to Alleviate Homesickness. in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.

Cited by

Sorted by citations

Authors
Han, B., Buchanan, G., & Mckay, D.
Affiliations
  • The University of Melbourne (AU)
  • RMIT University (AU)
ORCID
Venue
Proceedings of the 34th Australian Conference on Human-Computer Interaction
Publisher
ACM
Published
2022-11-29
Pages
9-21
DOI
10.1145/3572921.3572937
OpenAlex
W4362682549
Cited by
20
References
3
Citations by year
2026: 6 · 2025: 5 · 2024: 7 · 2023: 2
FWCI
3.13
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • Australian Conference on Human-Computer Interaction (OzCHI) (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Digital Education and Society
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
panopticon, student privacy, surveillance in higher education, student independence, speculative design, student identity, teacher-student relationship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review