CoChat
Sign Up
Cited by 14Ethics and Social Impacts of AIOnline Learning and AnalyticsArtificial Intelligence in Education

Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in Education

Open access
Tan, Mei, Lee, Hansol, Wang, Dakuo, Subramonyam, Hari·2024·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3637358
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite the promises of ML in education, its adoption in the classroom has surfaced numerous issues regarding fairness, accountability, and transparency, as well as concerns about data privacy and student consent. A root cause of these issues is the lack of understanding of the complex dynamics of education, including teacher-student interactions, collaborative learning, and classroom environments. To overcome these challenges and fully utilize the potential of ML in education, software practitioners need to work closely with educators and students to fully understand the context of the data (the backbone of ML applications) and collaboratively define the ML data specifications. To gain a deeper understanding of such a collaborative process, we conduct ten co-design sessions with ML software practitioners, educators, and students. In the sessions, teachers and students work with ML engineers, UX designers, and legal practitioners to define dataset characteristics for a given ML application. We find that stakeholders contextualize data based on their domain and procedural knowledge, proactively design data requirements to mitigate downstream harms and data reliability concerns, and exhibit role-based collaborative strategies and contribution patterns. Further, we find that beyond a seat at the table, meaningful stakeholder participation in ML requires structured supports: defined processes for continuous iteration and co-evaluation, shared contextual data quality standards, and information scaffolds for both technical and non-technical stakeholders to traverse expertise boundaries.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3637358

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Delgado et al. found that most AI projects bring stakeholders merely in as consultants, in one-off preference elicitation or UI decisions.

    “However, in a recent review paper, Delgado et al. [31] found that most AI projects bring stakeholders merely in as consultants, in one-off preference elicitation or UI decisions (cf. also [27, 108]).”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [108]

    Section 2.3 (page unavailable) says that, in educational AI, teachers and students are often consulted through need-finding interviews or product-design feedback and engaged only in a limited capacity as end users. It does not establish that most AI projects use one-off preference elicitation or UI decisions, and the cited paper is Tan et al.’s study, not a review by Delgado et al. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 72

  1. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 291-300.
  2. Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM Journal of Research and Development 63, 4/5 (2019), 6-1.
  3. Rob Ashmore, Radu Calinescu, and Colin Paterson. 2021. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys (CSUR) 54, 5 (2021), 1-39.
  4. Ryan S Baker and Aaron Hawn. 2022. Algorithmic bias in education. International Journal of Artificial Intelligence in Education 32, 4 (2022), 1052-1092.
  5. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021. The values encoded in machine learning research. arXiv preprint arXiv:2106.15590 (2021). · arXiv
  6. Melanie Birks, Ysanne Chapman, and Karen Francis. 2008. Memoing in qualitative research: Probing data and processes. Journal of research in nursing 13, 1 (2008), 68-75.
  7. Veronika Bogina, Alan Hartman, Tsvi Kuflik, and Avital Shulner-Tal. 2022. Educating software and AI stakeholders about algorithmic fairness, accountability, transparency and ethics. International Journal of Artificial Intelligence in Education 32, 3 (2022), 808-833.
  8. Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford. 2020. Overcoming failures of imagination in AI infused system development and deployment. arXiv preprint arXiv:2011.13416 (2020). · arXiv
  9. Amanda Buddemeyer, Erin Walker, and Malihe Alikhani. 2021. Words of Wisdom: Representational Harms in Learning From AI Communication. arXiv preprint arXiv:2111.08581 (2021). · arXiv
  10. Harshitha Challa, Nan Niu, and Reese Johnson. 2020. Faulty requirements made valuable: On the role of data quality in deep learning. In 2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE). IEEE, 61-69.
  11. Kasia S Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. 2022. The dataset nutrition label (2nd Gen): Leveraging context to mitigate harms in artificial intelligence. arXiv preprint arXiv:2201.03954 (2022). · arXiv
  12. Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2021. Stakeholder Participation in AI: Beyond" Add Diverse Stakeholders and Stir". arXiv preprint arXiv:2111.01122 (2021). · arXiv
  13. Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2342-2351.
  14. Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research (2020).
  15. Vladimir Estivill-Castro, Eugene Gilmore, and René Hexel. 2022. Constructing Explainable Classifiers from the Start-Enabling Human-in-the Loop Machine Learning. Information 13, 10 (2022), 464.
  16. Josh Gardner, Christopher Brooks, and Ryan Baker. 2019. Evaluating the fairness of predictive student models through slicing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge. 225-234.
  17. Philip Guo. 2013. Data science workflow: Overview and challenges. Commun. ACM (2013).
  18. Christina Harrington, Sheena Erete, and Anne Marie Piper. 2019. Deconstructing community-based collaborative design: Towards more equitable participatory design engagements. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-25.
  19. Amy K Heger, Liz B Marquis, Mihaela Vorvoreanu, Hanna Wallach, and Jennifer Wortman Vaughan. 2022. Un- derstanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1-29.
  20. Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2019. Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics 6, 2 (2019).
  21. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-16.
  22. Youyang Hou and Dakuo Wang. 2017. Hacking with NPOs: collaborative analytics and broker roles in civic data hackathons. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 1-16.
  23. Jessica Hullman, Sayash Kapoor, Priyanka Nanayakkara, Andrew Gelman, and Arvind Narayanan. 2022. The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning. arXiv preprint arXiv:2203.06498 (2022). · arXiv
  24. Hannah Kerner. 2020. Too many AI researchers think real-world problems are not relevant. Opinion. MIT Technology Review (2020).
  25. Mahmoud Keshavarz and Ramia Maze. 2013. Design and dissensus: framing and staging participation in design research. Design Philosophy Papers 11, 1 (2013), 7-29.
  26. René F. Kizilcec and Hansol Lee. 2022. Algorithmic Fairness in Education. In The Ethics of Artificial Intelligence in Education, W. Holmes & K. Porayska-Pomsta (Ed.). Routledge, Chapter 7.
  27. Laura Koesten, Kathleen Gregory, Paul Groth, and Elena Simperl. 2021. Talking datasets-understanding data sensemaking behaviours. International journal of human-computer studies 146 (2021), 102562.
  28. Matthijs Koopmans. 2020. Education is a complex dynamical system: Challenges for research. The Journal of Experimental Education 88, 3 (2020), 358-374.
  29. Sean Kross and Philip Guo. 2021. Orienting, framing, bridging, magic, and counseling: How data scientists navigate the outer loop of client collaborations in industry and academia. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-28.
  30. David Leslie, Michael Katell, Mhairi Aitken, Jatinder Singh, Morgan Briggs, Rosamund Powell, Cami Rincón, Antonella Perini, Smera Jayadeva, and Christopher Burr. 2022. Data Justice in Practice: A Guide for Developers. arXiv preprint arXiv:2205.01037 (2022). · arXiv
  31. Warren Li, Kaiwen Sun, Florian Schaub, and Christopher Brooks. 2022. Disparities in students' propensity to consent to learning analytics. International Journal of Artificial Intelligence in Education 32, 3 (2022), 564-608.
  32. Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. 2022. Advances, challenges and opportunities in creating data for trustworthy AI. Nature Machine Intelligence 4, 8 (2022), 669-677.
  33. Yaoli Mao, Dakuo Wang, Michael Muller, Kush R Varshney, Ioana Baldini, Casey Dugan, and Aleksandra Mojsilović. 2019. How data scientistswork together with domain experts in scientific collaborations: To find the right answer or to ask the right question? Proceedings of the ACM on Human-Computer Interaction 3, GROUP (2019), 1-23.
  34. Mirko Marras, Ludovico Boratto, Guilherme Ramos, and Gianni Fenu. 2022. Equality of learning opportunity via individual fairness in personalized recommendations. International Journal of Artificial Intelligence in Education 32, 3 (2022), 636-684.
  35. Konstantinos Michos, Charles Lang, Davinia Hernández-Leo, and Detra Price-Dennis. 2020. Involving teachers in learning analytics design: Lessons learned from two case studies. In Proceedings of the Tenth international conference on learning analytics & knowledge. 94-99.
  36. Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-15.
  37. Michael Muller and Angelika Strohmayer. 2022. Forgetting Practices in the Data Sciences. In CHI Conference on Human Factors in Computing Systems. 1-19.
  38. Aiswarya Raj Munappy, Jan Bosch, and Helena Homström Olsson. 2020. Data pipeline management in practice: Challenges and opportunities. In International Conference on Product-Focused Software Process Improvement. Springer, 168-184.
  39. Tadhg Nagle, Thomas C Redman, and David Sammon. 2017. Only 3% of companies' data meets basic quality standards. Harvard Business Review 95, 5 (2017), 2-5.
  40. Jaclyn Ocumpaugh, Ryan Baker, Sujith Gowda, Neil Heffernan, and Cristina Heffernan. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45 (05 2014). https://doi.org/10.1111/bjet.12156 · DOI · Paper page
  41. Soya Park, April Yi Wang, Ban Kawas, Q Vera Liao, David Piorkowski, and Marina Danilevsky. 2021. Facilitating knowledge sharing from domain experts to data scientists for building nlp models. In 26th International Conference on Intelligent User Interfaces. 585-596.
  42. Samir Passi and Phoebe Sengers. 2020. Making data science systems work. Big Data & Society 7, 2 (2020), 2053951720939605.
  43. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021), 100336.
  44. Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2017. Data management challenges in production machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data. 1723-1726.
  45. Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2018. Data lifecycle challenges in production machine learning: a survey. ACM SIGMOD Record 47, 2 (2018), 17-28.
  46. Isak Potgieter. 2020. Privacy concerns in educational data mining and learning analytics. The International Review of Information Ethics 28 (2020).
  47. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. arXiv preprint arXiv:2204.01075 (2022). · arXiv
  48. Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-23.
  49. John T Richards, David Piorkowski, Michael Hind, Stephanie Houde, Aleksandra Mojsilovic, and Kush R Varshney. 2021. A Human-Centered Methodology for Creating AI FactSheets. IEEE Data Eng. Bull. 44, 4 (2021), 47-58.
  50. Rashida Richardson, Jason M Schultz, and Kate Crawford. 2019. Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. NYUL Rev. Online 94 (2019), 15.
  51. Ido Roll and Ruth Wylie. 2016. Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education 26, 2 (2016), 582-599.
  52. Daniela K Rosner, Saba Kawas, Wenqi Li, Nicole Tilly, and Yi-Chen Sung. 2016. Out of time, out of place: Reflections on design workshops as a research method. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1131-1141.
  53. Jeffrey S Saltz and Nancy W Grady. 2017. The ambiguity of data science team roles and the need for a data science workforce framework. In 2017 IEEE international conference on big data (Big Data). IEEE, 2355-2361.
  54. Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. CoDesign 4, 1 (2008), 5-18. https://doi.org/10.1080/15710880701875068 arXiv:https://doi.org/10.1080/15710880701875068 · DOI · Paper page
  55. Daniel Schiff. 2021. Out of the laboratory and into the classroom: the future of artificial intelligence in education. AI & society 36, 1 (2021), 331-348.
  56. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015).
  57. Zheyuan Ryan Shi, Claire Wang, and Fei Fang. 2020. Artificial intelligence for social good: A survey. arXiv preprint arXiv:2001.01818 (2020). · arXiv
  58. Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2020. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423 (2020). · arXiv
  59. Marc Steen. 2013. Co-Design as a Process of Joint Inquiry and Imagination. Design Issues 29, 2 (04 2013), 16-28. https://doi.org/10.1162/DESI_a_00207 arXiv:https://direct.mit.edu/desi/article-pdf/29/2/16/1715163/desi_a_00207.pdf · DOI
  60. Hariharan Subramonyam, Jane Im, Colleen Seifert, and Eytan Adar. 2022. Solving Separation-of-Concerns Problems in Collaborative Design of Human-AI Systems through Leaky Abstractions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 481, 21 pages. https://doi.org/10.1145/3491102.3517537 · DOI
  61. Hariharan Subramonyam, Jane Im, Colleen Seifert, and Eytan Adar. 2022. Solving Separation-of-Concerns Problems in Collaborative Design of Human-AI Systems through Leaky Abstractions. In CHI Conference on Human Factors in Computing Systems. 1-21.
  62. Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Towards a process model for co-creating AI experiences. In Designing Interactive Systems Conference 2021. 1529-1543.
  63. Hariharan Subramonyam, Colleen Seifert, and MI Eytan Adar. 2021. How Can Human-Centered Design Shape Data- Centric AI?. In NeurIPS Data-Centric AI Workshop. Retrieved from https://haridecoded. com/resources/AIX_NeurIPS_2021. pdf.
  64. Alex S Taylor, Siân Lindley, Tim Regan, David Sweeney, Vasillis Vlachokyriakos, Lillie Grainger, and Jessica Lingel. 2015. Data-in-place: Thinking through the relations between data and community. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2863-2872.
  65. Nenad Tomašev, Julien Cornebise, Frank Hutter, Shakir Mohamed, Angela Picciariello, Bec Connelly, Danielle Belgrave, Daphne Ezer, Fanny Cachat van der Haert, Frank Mugisha, et al. 2020. AI for social good: unlocking the opportunity for positive impact. Nature Communications 11, 1 (2020), 1-6.
  66. Janet Vertesi and Paul Dourish. 2011. The value of data: considering the context of production in data economies. In Proceedings of the ACM 2011 conference on Computer supported cooperative work. 533-542.
  67. April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2022. Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks. ACM Transactions on Computer-Human Interaction 29, 2 (2022), 1-33.
  68. Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. 2023. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal (2023), 1-23.
  69. Marian G Williams and Vivienne Begg. 1993. Translation between software designers and users. Commun. ACM 36, 6 (1993), 102-103.
  70. Amy X Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? roles, workflows, and tools. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1-23.
  71. Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022. StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. In CHI Conference on Human Factors in Computing Systems. 1-21.
  72. Qi Zhou, Wannapon Suraworachet, Stanislav Pozdniakov, Roberto Martinez-Maldonado, Tom Bartindale, Peter Chen, Dan Richardson, and Mutlu Cukurova. 2021. Investigating students' experiences with collaboration analytics for remote group meetings. In International Conference on Artificial Intelligence in Education. Springer, 472-485.

Cited by

Sorted by citations

  • Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation

    Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1 · see citing works

    “…our approach to providing disability advocacy organizations with technology scaffolds and facilitating opportunities for community dialogue, which enabled the production of nuanced, contextual accounts of what ‘good’ representation means for each community, and its embedding within data for AI.”
    Influential

Authors
Tan, M., Lee, H., Wang, D., & Subramonyam, H.
Affiliations
  • Stanford University (US)
  • Northeastern University (US)
  • Stanford University, Stanford, CA, USA
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2024-04-17
Volume
8
Issue
CSCW1
Pages
1-32
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3637358
OpenAlex
W4396218892
Cited by
14
References
14
Citations by year
2026: 5 · 2025: 5 · 2024: 4
FWCI
4.51
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
machine learning in education, stakeholder co-design, teacher-student collaboration, data requirements, data quality, fairness and accountability, data privacy, student consent, cooperative learning, role-based collaboration, educational technology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review