Co-designing AI with youth partners: Enabling ideal classroom relationships through a novel AI relational privacy ethical framework
Open accessAbstract
In recent years, the design of AI-based tools for educational spaces have been largely driven by researchers who impart their past expertises, experiences, and perspectives in the design process. While this typically leads to technically feasible designs and are often well-grounded in theories of learning, youth agency is typically limited in this process. In this paper, we argue that designers have a significant ethical responsibility to incorporate youth voices – in particular, their dreams and concerns – into the design of AI tools starting from conception. This need is particularly important as new applications for AI, such as AI-supported collaboration, introduce new surveillance vectors into classroom spaces. Drawing from recent scholarship which advances ethics and relationality in participatory co-design with youth, we introduce a co-design methodology in which youth are supported in imagining expansive technical possibilities for K-12 public schools, grounded within affordances, limitations, and tradeoffs of AI and machine learning techniques . This approach is demonstrated through our Learning Futures Workshop , which brought together 30 historically minoritized youth in conversation with experts in both education and technology. Through detailed case study on the enactment of the workshop, including a thematic analysis of the activities the youth engaged in and their outputs, we identified new, expansive relational possibilities for AI, ethical commitments to support the design, and finally, developed a novel AI Relational Privacy ethical framework that supports the design of new collaborative AI platforms. We conclude by connecting these findings and frameworks to the design of newly enacted AI-based applications and underlying data infrastructures.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The students' distinction between immediate classroom and larger school aligns with the concept of relational privacy.
“which aligns with the concept of 'relational privacy' (Chang et al., 2025).”
From A Feasibility and Implementation Integrity Study of the Community Builder (CoBi): An AI-based Collaboration Support System in K-12 Classrooms (2025) · cites this paper as Chang et al., 2025
Cites
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · see citing works
Race after technology. Abolitionist tools for the new Jim Code
Merz, S. (2020) · Ethnic and Racial Studies · doi:10.1080/01419870.2020.1715454 · Cited by 1,889 · see citing works
Artificial intelligence in education: Addressing ethical challenges in K-12 settings
Akgun, S., & Greenhow, C. (2022) · AI and Ethics · doi:10.1007/s43681-021-00096-7 · Cited by 1,473 · see citing works
Ethics of AI in Education: Towards a Community-Wide Framework
Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., et al. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00239-1 · Cited by 1,437 · see citing works
Ethical principles for artificial intelligence in education
Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11316-w · Cited by 1,228 · see citing works
Chiu, T. K. F., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100118 · Cited by 1,181 · see citing works
The "Grammar" of Schooling: Why Has it Been so Hard to Change?
Tyack, D., & Tobin, W. (1994) · American Educational Research Journal · doi:10.3102/00028312031003453 · Cited by 1,169 · see citing works
Learning Analytics: Ethical Issues and Dilemmas
Slade, S., & Prinsloo, P. (2013) · American Behavioral Scientist · doi:10.1177/0002764213479366 · Cited by 1,078 · see citing works
The role of children in the design of new technology
(2002) · Behaviour & Information Technology · doi:10.1080/01449290110108659 · Cited by 1,071 · see citing works
Research–Practice Partnerships in Education: Outcomes, Dynamics, and Open Questions
Coburn, C. E., & Penuel, W. R. (2016) · Educational Researcher · doi:10.3102/0013189x16631750 · Cited by 939 · see citing works
Privacy as Contextual IntegrityPreprint
Nissenbaum, H. F. (2004) · SSRN Electronic Journal · Cited by 897 · see citing works
Druin, A. (1999) · doi:10.1145/302979.303166 · Cited by 856 · see citing works
Gutiérrez, K. D., Rymes, B., & Larson, J. (1995) · Harvard Educational Review · doi:10.17763/haer.65.3.r16146n25h4mh384 · Cited by 771 · see citing works
Anderson, J. R., Boyle, C., & Reiser, B. (1985) · Science · doi:10.1126/science.228.4698.456 · Cited by 699 · see citing works
Bang, M., & Vossoughi, S. (2016) · Cognition and Instruction · doi:10.1080/07370008.2016.1181879 · Cited by 628 · see citing works
Race after technology Abolitionist tools for the new Jim code
Benjamin, R. 1. (2019) · Cited by 578 · see citing works
Ethical and privacy principles for learning analytics
Pardo, A., & Siemens, G. (2014) · British Journal of Educational Technology · doi:10.1111/bjet.12152 · Cited by 577 · see citing works
Six Human-Centered Artificial Intelligence Grand Challenges
Ozmen Garibay, O., Winslow, B., Andolina, S., Antona, M., Bodenschatz, A., Coursaris, C., Falco, G., Fiore, S. M., Garibay, I., Grieman, K., et al. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2153320 · Cited by 501 · see citing works
Social Design Experiments: Toward Equity by Design
Gutiérrez, K. D., & Jurow, A. S. (2016) · Journal of the Learning Sciences · doi:10.1080/10508406.2016.1204548 · Cited by 422 · see citing works
Re-mediating Literacy: Culture, Difference, and Learning for Students From Nondominant Communities
Gutiérrez, K. D., Morales, P. Z., & Martínez, D. C. (2009) · Review of Research in Education · doi:10.3102/0091732x08328267 · Cited by 387 · see citing works
Drachsler, H., & Greller, W. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883893 · Cited by 374 · see citing works
Xu, W., Dainoff, M. J., Ge, L., & Gao, Z. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2041900 · Cited by 363 · see citing works
Systematic Review: Trust-Building Factors and Implications for Conversational Agent Design
Rheu, M., Shin, J. Y., Peng, W., & Huh-Yoo, J. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1807710 · Cited by 360 · see citing works
Student perceptions of privacy principles for learning analytics
Ifenthaler, D., & Schumacher, C. (2016) · Educational Technology Research and Development · doi:10.1007/s11423-016-9477-y · Cited by 314 · see citing works
Guha, M. L., Druin, A., & Fails, J. A. (2013) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2012.08.003 · Cited by 307 · see citing works
Ethical principles for artificial intelligence in K-12 education
Adams, C., Pente, P., Lemermeyer, G., & Rockwell, G. (2023) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2023.100131 · Cited by 288 · see citing works
A Holistic Approach to the Design of Artificial Intelligence (AI) Education for K-12 Schools
Chiu, T. K. F. (2021) · TechTrends · doi:10.1007/s11528-021-00637-1 · Cited by 271 · see citing works
The Learning Styles Myth is Thriving in Higher Education
Newton, P. M. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01908 · Cited by 219 · see citing works
Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms
Lin, P., & Brummelen, J. V. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445377 · Cited by 195 · see citing works
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-23204-7_14 · Cited by 189 · see citing works
Intelligent tutors as teachers' aides
Holstein, K., McLaren, B. M., & Aleven, V. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027451 · Cited by 179 · see citing works
Dwyer, K. K., Bingham, S. G., Carlson, R. E., Prisbell, M., Cruz, A. M., & Fus, D. A. (2004) · Communication Research Reports · doi:10.1080/08824090409359988 · Cited by 154 · see citing works
Hrastinski, S., Olofsson, A. D., Arkenback, C., Ekström, S., Ericsson, E., Fransson, G., Jaldemark, J., Ryberg, T., Öberg, L.-M., Fuentes, A., et al. (2019) · Postdigital Science and Education · doi:10.1007/s42438-019-00046-x · Cited by 145 · see citing works
Practical ethics for building learning analytics
Kitto, K., & Knight, S. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12868 · Cited by 139 · see citing works
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170377 · Cited by 129 · see citing works
Participatory Design for Learning
(2017) · Routledge eBooks · doi:10.4324/9781315630830 · Cited by 127 · see citing works
Student privacy self-management: implications for learning analytics
Prinsloo, P., & Slade, S. (2015) · Proceedings of the Fifth International Conference on Learning Analytics And Knowledge · doi:10.1145/2723576.2723585 · Cited by 119 · see citing works
The use of Artificial intelligence in school science: a systematic literature review
Heeg, D. M., & Avraamidou, L. (2023) · Educational Media International · doi:10.1080/09523987.2023.2264990 · Cited by 118 · see citing works
Touretzky, D. S., Gardner‐McCune, C., Breazeal, C., Martin, F., & Seehorn, D. (2019) · AI Magazine · doi:10.1609/aimag.v40i4.5289 · Cited by 112 · see citing works
Learning analytics at the intersections of student trust, disclosure and benefit
Slade, S., Prinsloo, P., & Khalil, M. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303796 · Cited by 104 · see citing works
Sun, C., Shute, V. J., Stewart, A. E. B., Beck-White, Q., Reinhardt, C. R., Zhou, G., Duran, N., & D'Mello, S. K. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107120 · Cited by 100 · see citing works
Self-citationAslan, S., Alyüz, N., Tanriover, C., Mete, S. E., Okur, E., D’Mello, S. K., & Esme, A. A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300534 · Cited by 98 · see citing works
Self-citationCo-designing online privacy-related games and stories with children
Kumar, P. C., Jessica Vitak, B., Chetty, M., Clegg, T., Yang, J., McNally, B., & Bonsignore, E. (2018) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3202185.3202735 · Cited by 95 · see citing works
Lawrence, L., Echeverría, V., Yang, K. B., Aleven, V., & Rummel, N. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13372 · Cited by 88 · see citing works
Cerratto Pargman, T., & McGrath, C. (2021) · Journal of Learning Analytics · doi:10.18608/jla.2021.1 · Cited by 88 · see citing works
Cultural Tensions in the Scripted Classroom
Gutiérrez, K. D., Larson, J., & Kreuter, B. (1995) · Urban Education · doi:10.1177/0042085995029004004 · Cited by 80 · see citing works
Brion‐Meisels, G., & Alter, Z. (2018) · Harvard Educational Review · doi:10.17763/1943-5045-88.4.429 · Cited by 63 · see citing works
Intelligent Agent–Supported Online Education
Li, X. (2007) · Decision Sciences Journal of Innovative Education · doi:10.1111/j.1540-4609.2007.00143.x · Cited by 52 · see citing works
Exploring the role of off-task activity on students’ collaborative dynamics.
Langer-Osuna, J. M., Gargroetzi, E. C., Munson, J., & Chavez, R. (2020) · Journal of Educational Psychology · doi:10.1037/edu0000464 · Cited by 43 · see citing works
Vossoughi, S., & Escudé, M. (2016) · Anthropology & Education Quarterly · doi:10.1111/aeq.12134 · Cited by 42 · see citing works
Intentionally Addressing Nested Systems of Power in Schooling through Teacher Solidarity Co-Design
Philip, T. M., Pham, J. H., Scott, M., & Cortez, A. (2022) · Cognition and Instruction · doi:10.1080/07370008.2021.2010208 · Cited by 42 · see citing works
Self-citationTanksley, T. C. (2024) · English Teaching: Practice & Critique · doi:10.1108/etpc-08-2023-0102 · Cited by 32 · see citing works
VanLehn, K. A., Burkhardt, H., Cheema, S., Kang, S., Pead, D., Schoenfeld, A. H., & Wetzel, J. (2019) · Interactive Learning Environments · doi:10.1080/10494820.2019.1616567 · Cited by 30 · see citing works
Chang, M. A., Wong, R. Y., Breideband, T., Philip, T. M., McKoy, A., Cortez, A., & D'Mello, S. K. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642559 · Cited by 28 · see citing works
Self-citationCo-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics
Ahn, J., Campos, F., Nguyen, H., Hays, M., & Morrison, J. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448145 · Cited by 28 · see citing works
Brummelen, J. V., Kelleher, M., Tian, M. C., & Nguyen, N. H. (2023) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3585088.3589353 · Cited by 22 · see citing works
Breideband, T., Bush, J. B., Chandler, C., Chang, M. A., Dickler, R., Foltz, P. W., Ganesh, A., Lieber, R., Penuel, W. R., Reitman, J. G., et al. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3607498 · Cited by 22 · see citing works
Self-citationD’Mello, S. K., Biddy, Q. L., Breideband, T., Bush, J. B., Chang, M. A., Cortez, A., Flanigan, J., Foltz, P. W., Gorman, J. C., Hirshfield, L., et al. (2024) · AI Magazine · doi:10.1002/aaai.12158 · Cited by 18 · see citing works
Self-citationChild-Computer Interaction SIG
Hourcade, J. P., Revelle, G. L., Zeising, A., Iversen, O. S., Parés, N., Bekker, T. M., & Read, J. C. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2886433 · Cited by 18 · see citing works
Brummelen, J. V., Tian, M. C., Kelleher, M., & Nguyen, N. H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26883 · Cited by 17 · see citing works
CPSCoach: The Design and Implementation of Intelligent Collaborative Problem Solving Feedback
Stewart, A. E. B., Rao, A., Michaels, A., Sun, C., Duran, N. D., Shute, V. J., & D’Mello, S. K. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-36272-9_58 · Cited by 11 · see citing works
Self-citationFunds of Identity in Co-Design
Iriarte, M. P., Koren, N., Blinder, E. B., & Bonsignore, E. (2023) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3585088.3593901 · Cited by 6 · see citing works
Lawrence, L., Guo, B., Yang, K., Echeverría, V., Kang, Z., Bathala, V., & Li, C. (2022) · Digital Commons - USU (Utah State University) · Cited by 5 · see citing works
Child-computer interaction SIG:New challenges and opportunities
Hourcade, J. P., Revelle, G. L., Zeising, A., Iversen, O. S., Parés, N., Bekker, M. M. =. T., & Read, J. C. (2016) · TU/e Research Portal · Cited by 4 · see citing works
Towards Bringing Human-Centered Design to K-12 and Post-Secondary Education
Shehab, S., Tissenbaum, M., Lawrence, L., Lewis, D. R., Easterday, M. W., Carlson, S. E., Royalty, A., Chen, H. L., Sheppard, S., Goldman, S. V., et al. (2021) · Digital Commons - USU (Utah State University) · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
4 of the 69 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Fostering responsible AI literacy: A systematic review of K-12 AI ethics education
Ma, M., Ng, D. T. K., Liu, Z., & Wong, G. K. W. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100422 · Cited by 78 · see citing works
Generative Artificial Intelligence (AI) in education: from organizing visions to official guidelines
Nguyen, A., Kishore, S., Hong, Y., Qutab, S., & Dang, B. (2025) · Information Technology and People · doi:10.1108/itp-08-2024-1026 · Cited by 24 · see citing works
Zhong, Y., & Rosli, M. S. B. (2025) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v14-i3/25813 · Cited by 13 · see citing works
Breideband, T., Bush, J. B., Reitman, J. G., Rose, S., Weatherley, J., Penuel, W. R., & D’Mello, S. K. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00509-2 · Cited by 5 · see citing works
“…their immediate classroom where they felt comfortable sharing CoBi results with the teacher and the larger school where they felt more discomfort for the prospect of their data being shared with the principal’s office, which aligns with the concept of “relational pri-vacy” (Chang et al., 2025).”
Self-citationEngineering Human–AI Leadership for School Equity
Ismail, M. A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0388-6.ch003 · Cited by 4 · see citing works
Interaction Design Education: AI Usage in the Design Process
Spérano, I., Roberge, J., McDow-York, D., Felfly, I., & Santos, H. A. (2025) · The International Journal of Design Education · doi:10.18848/2325-128x/cgp/v19i02/229-246 · Cited by 3 · see citing works
Beyond the lesson plan: Reflective practice as a catalyst for teacher professional growth
Harsono, A. M. B., Mahmudah, F. N., Putra, E. C. S., Rahman, A., Dewantara, B. A., Arsyad, M. Z. T., & Alfarisa, F. (2025) · Priviet Social Sciences Journal · doi:10.55942/pssj.v5i11.698 · Cited by 2 · see citing works
Developing an Ethical Framework for Youth
Ahrari, S., & Al-Issa, R. S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-2170-7.ch010 · Cited by 1 · see citing works
Darabipourshiraz, H., Smyslova, D., Xu, D., Jiang, S., & Long, D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791555 · Cited by 1 · see citing works
Lo, S., Lea, T., Mavromoustakos, A., Joseph, B., Yii, M., Exintaris, B., Karunaratne, N., Pearson, D. L., Ritchie, T., Yuriev, E., et al. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1818781
Responsible AI in Education: Multilingual, Multimodal, and Relational Futures
Lee, H. H. (2026) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/01614681261493389
Zaruma-Game, J., Criollo-Córdova, I., Marin-Molina, J., Sornoza-Vera, A., Saraguro-Bravo, R., & Paderewski, P. (2026) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg70169.2026.11695432
Recentring young people in AI for mental health support: breaking the evidence-design feedback loop
Sangha, J. K., Partridge, S. R., & Raeside, R. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1954336
- Authors
- Chang, M. A., Tissenbaum, M., Philip, T. M., & D’Mello, S. K.
- Affiliations
- Boston University (US)
- University of Illinois Urbana-Champaign (US)
- University of California, Berkeley (US)
- University of Colorado Boulder (US)
- Corresponding author
- Chang, Michael Alan
- Venue
- Computers and Education: Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2025-06
- Volume
- 8
- Pages
- 100364
- ISSN
- 2666-920X (ISSN-L)
- OpenAlex
- W4406411734
- Cited by
- 12
- References
- 69
- Citations by year
- 2026: 6 · 2025: 6
- FWCI
- 12.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Science Foundation (2019805, 2021159)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Digital Education and Society
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- youth participation, participatory design, classroom relationships, K-12 education, AI ethics, relationality, data infrastructure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
