CoChat
Sign Up
Cited by 67Ethics and Social Impacts of AIGame Theory and Voting SystemsSchool Choice and Performance

Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms

Open access
Robertson, Samantha, Nguyen, Tonya, Salehi, Niloufar·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3411764.3445748
Open this paper in CoChat — free accountOpen PDF

Abstract

Across the United States, a growing number of school districts are turning to matching algorithms to assign students to public schools. The designers of these algorithms aimed to promote values such as transparency, equity, and community in the process. However, school districts have encountered practical challenges in their deployment. In fact, San Francisco Unified School District voted to stop using and completely redesign their student assignment algorithm because it was frustrating for families and it was not promoting educational equity in practice. We analyze this system using a Value Sensitive Design approach and find that one reason values are not met in practice is that the system relies on modeling assumptions about families’ priorities, constraints, and goals that clash with the real world. These assumptions overlook the complex barriers to ideal participation that many families face, particularly because of socioeconomic inequalities. We argue that direct, ongoing engagement with stakeholders is central to aligning algorithmic values with real world conditions. In doing so we must broaden how we evaluate algorithms while recognizing the limitations of purely algorithmic solutions in addressing complex socio-political problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3411764.3445748

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 54

  1. Atila Abdulkadiroglu, Parag A Pathak, Jonathan Schellenberg, and Christopher R Walters. 2017. Do Parents Value School Efectiveness? Working Paper 23912. National Bureau of Economic Research. https://doi.org/10.3386/w23912 · DOI
  2. Atila Abdulkadiroğlu and Tayfun Sönmez. 2003. School Choice: A Mechanism Design Approach. American Economic Review 93, 3 (June 2003), 729-747. https: //doi.org/10.1257/000282803322157061 · DOI
  3. Morgan G. Ames. 2016. Learning consumption: Media, literacy, and the legacy of One Laptop per Child. The Information Society 32, 2 (2016), 85-97. https: //doi.org/10.1080/01972243.2016.1130497 · DOI
  4. Eric PS Baumer. 2017. Toward human-centered algorithm design. Big Data & Society 4, 2 (2017), 1-12. https://doi.org/10.1177/2053951717718854 · DOI · Paper page
  5. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS'16).
  6. Alan Borning and Michael Muller. 2012. Next Steps for Value Sensitive Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 1125-1134. https://doi.org/10.1145/2207676.2208560 · DOI · Paper page
  7. G. E. P. Box. 1979. Robustness in the Strategy of Scientifc Model Building. In Robustness in Statistics, Robert L. Launder and Graham N. Wilkinson (Eds.). Academic Press, Cambridge, MA, USA, 201 -236. https://doi.org/10.1016/B978- 0-12-438150-6.50018-2 · DOI
  8. Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew To- bin, and Rhema Vaithianathan. 2019. Toward Algorithmic Accountability in Public Services: A Qualitative Study of Afected Community Perspectives on Algorithmic Decision-Making in Child Welfare Services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, Article 41, 12 pages. https://doi.org/10.1145/3290605.3300271 · DOI · Paper page
  9. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classifcation. In Proceedings of the 1st Confer- ence on Fairness, Accountability and Transparency (Proceedings of Machine Learn- ing Research), Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 77-91. http://proceedings.mlr.press/v81/buolamwini18a.html
  10. Simon Burgess, Ellen Greaves, Anna Vignoles, and Deborah Wilson. 2015. What Parents Want: School Preferences and School Choice. The Economic Journal 125, 587 (2015), 1262-1289. https://doi.org/10.1111/ecoj.12153 · DOI
  11. Stevie Chancellor, Eric P. S. Baumer, and Munmun De Choudhury. 2019. Who is the "Human" in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 147 (Nov. 2019), 32 pages. https://doi.org/10.1145/3359249 · DOI
  12. Kathy Charmaz. 2014. Constructing grounded theory: A practical guide through qualitative research (2 ed.). SAGE Publications Ltd, London, United Kingdom.
  13. Sasha Costanza-Chock. 2018. Design Justice: Towards an Intersectional Feminist Framework for Design Theory and Practice. In Proceedings of the Design Research Society. Design Research Society, London, United Kingdom, 14. https://ssrn.com/ abstract=3189696
  14. Kate Crawford. 2016. Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics. Science, Technology, & Human Values 41, 1 (2016), 77-92. https://doi.org/10.1177/0162243915589635 · DOI
  15. Janet Davis and Lisa P. Nathan. 2015. Value Sensitive Design: Applications, Adaptations, and Critiques. In Handbook of Ethics, Values, and Technological Design, Jeroen van den Hoven, Pieter E. Vermaas, and Ibo van de Poel (Eds.). Springer, Dordrecht, Netherlands, 11-40.
  16. Thomas Dietz, Elinor Ostrom, and Paul C. Stern. 2003. The Struggle to Govern the Commons. Science 302, 5652 (2003), 1907-1912. https://doi.org/10.1126/ science.1091015 · DOI · Paper page
  17. Tingting Ding and Andrew Schotter. 2017. Matching and chatting: An exper- imental study of the impact of network communication on school-matching mechanisms. Games and Economic Behavior 103 (2017), 94 -115. https: //doi.org/10.1016/j.geb.2016.02.004 John Nash Memorial. · DOI
  18. Tingting Ding and Andrew Schotter. 2019. Learning and Mechanism Design: An Experimental Test of School Matching Mechanisms with Intergenerational Advice. The Economic Journal 129, 623 (05 2019), 2779-2804.
  19. Eve L. Ewing. 2018. Ghosts in the Schoolyard: Racism and School Closings on Chicago's South Side. University of Chicago Press, Chicago, IL, USA.
  20. Mary Flanagan, Daniel C. Howe, and Helen Nissenbaum. 2005. Values at Play: Design Tradeofs in Socially-Oriented Game Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI '05). Association for Computing Machinery, New York, NY, USA, 751-760. https://doi.org/10.1145/1054972.1055076 · DOI
  21. Batya Friedman, David G. Hendry, and Alan Borning. 2017. A Survey of Value Sen- sitive Design Methods. Foundations and Trends in Human-Computer Interaction 11, 2 (2017), 63-125. https://doi.org/10.1561/1100000015 · DOI · Paper page
  22. Batya Friedman and Peter H. Kahn Jr. 2003. Human Values, Ethics, and Design. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, Jacko JA Sears A (Ed.). L. Erlbaum Associates Inc., USA, 1177-1201.
  23. B Friedman, PH Jr Kahn, and A Borning. 2006. Value Sensitive Design and Information Systems. In Human-Computer Interaction in Management Information Systems: Foundations, Ben Shneiderman, Ping Zhang, and Dennis Galletta (Eds.).
  24. Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L. Beam, Irene Ya-Ping Chen, and Rajesh Ranganath. 2020. A Review of Challenges and Oppor- tunities in Machine Learning for Health.. In AMIA Joint Summits on Translational Science. American Medical Informatics Association, USA, 191-200.
  25. Steven Glazerman and Dallas Dotter. 2017. Market Signals: Evidence on the Determinants and Consequences of School Choice From a Citywide Lottery. Educational Evaluation and Policy Analysis 39, 4 (2017), 593-619. https://doi.org/ 10.3102/0162373717702964 · DOI
  26. Yannai A. Gonczarowski, Noam Nisan, Lior Kovalio, and Assaf Romm. 2019. Matching for the Israeli "Mechinot" Gap-Year Programs: Handling Rich Diversity Requirements. In Proceedings of the 2019 ACM Conference on Economics and Computation (Phoenix, AZ, USA) (EC '19). Association for Computing Machinery, New York, NY, USA, 321. https://doi.org/10.1145/3328526.3329620 · DOI
  27. Pablo Guillen and Rustamdjan Hakimov. 2017. Not quite the best response: truth- telling, strategy-proof matching, and the manipulation of others. Experimental Economics 20, 3 (2017), 670-686.
  28. Pablo Guillen and Rustamdjan Hakimov. 2018. The efectiveness of top-down advice in strategy-proof mechanisms: A feld experiment. European Economic Review 101, C (2018), 505-511.
  29. Pablo Guillen and Alexander Hing. 2014. Lying through their teeth: Third party advice and truth telling in a strategy proof mechanism. European Economic Review 70 (2014), 178 -185. https://doi.org/10.1016/j.euroecorev.2014.05.002 · DOI
  30. Isa E. Hafalir, M. Bumin Yenmez, and Muhammed A. Yildirim. 2013. Efective afrmative action in school choice. Theoretical Economics 8, 2 (2013), 325-363. https://doi.org/10.3982/TE1135 · DOI
  31. Christina Harrington, Sheena Erete, and Anne Marie Piper. 2019. Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 216 (Nov. 2019), 25 pages. https://doi.org/10.1145/3359318 · DOI · Paper page
  32. Avinatan Hassidim, Assaf Romm, and Ran I. Shorrer. 2016. "Strategic" Behavior in a Strategy-Proof Environment. In Proceedings of the 2016 ACM Conference on Economics and Computation (Maastricht, The Netherlands) (EC '16). Association for Computing Machinery, New York, NY, USA, 763-764. https://doi.org/10. 1145/2940716.2940751 · DOI
  33. Justine S. Hastings, Thomas J. Kane, and Douglas O. Staiger. 2007. Heterogeneous Preferences and the Efcacy of Public School Choice. Working Paper 12145. National Bureau of Economic Research. https://doi.org/10.3386/w12145 · DOI
  34. Zoë Hitzig. 2020. The normative gap: Mechanism design and ideal theories of justice. Economics and Philosophy 36, 3 (2020), 407-434. https://doi.org/10.1017/ S0266267119000270 · DOI
  35. Adam J. Kapor, Christopher A. Neilson, and Seth D. Zimmerman. 2020. Hetero- geneous Beliefs and School Choice Mechanisms. American Economic Review 110, 5 (May 2020), 1274-1315. https://doi.org/10.1257/aer.20170129 · DOI
  36. T. Kulesza, S. Stumpf, M. Burnett, S. Yang, I. Kwan, and W. Wong. 2013. Too much, too little, or just right? Ways explanations impact end users' mental models. In 2013 IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, San Jose, CA, USA, 3-10. https://doi.org/10.1109/VLHCC.2013.6645235 · DOI · Paper page
  37. Christopher A. Le Dantec, Erika Shehan Poole, and Susan P. Wyche. 2009. Values as Lived Experience: Evolving Value Sensitive Design in Support of Value Dis- covery. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI '09). Association for Computing Machinery, New York, NY, USA, 1141-1150. https://doi.org/10.1145/1518701.1518875 · DOI
  38. Min Kyung Lee and Su Baykal. 2017. Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 1035-1048. https: //doi.org/10.1145/2998181.2998230 · DOI · Paper page
  39. Min Kyung Lee, Anuraag Jain, Hea Jin Cha, Shashank Ojha, and Daniel Kusbit. 2019. Procedural Justice in Algorithmic Fairness: Leveraging Transparency and Outcome Control for Fair Algorithmic Mediation. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 182 (Nov. 2019), 26 pages. https://doi.org/10.1145/ · DOI
  40. Min Kyung Lee, Ji Tae Kim, and Leah Lizarondo. 2017. A Human-Centered Approach to Algorithmic Services: Considerations for Fair and Motivating Smart Community Service Management That Allocates Donations to Non-Proft Organi- zations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machin- ery, New York, NY, USA, 3365-3376. https://doi.org/10.1145/3025453.3025884 · DOI
  41. Noëmi Manders-Huits. 2011. What Values in Design? The Challenge of Incorpo- rating Moral Values into Design. Science and Engineering Ethics 17, 2 (June 2011), 271-287. https://doi.org/10.1007/s11948-010-9198-2 · DOI
  42. Thành Nguyen and Rakesh Vohra. 2019. Stable Matching with Proportionality Constraints. Operations Research 67, 6 (2019), 1503-1519. https://doi.org/10.1287/ opre.2019.1909 · DOI
  43. Helen Nissenbaum. 2011. A Contextual Approach to Privacy Online. Daedalus 140, 4 (2011), 32-48.
  44. Safya Umoja Noble. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism. New York University Press, New York, NY, USA.
  45. Hessel Oosterbeek, Sándor Sóvágó, and Bas Klaauw. 2019. Why are Schools Segregated? Evidence from the Secondary-School Match in Amsterdam. Discussion Paper DP13462. Centre for Economic Policy Research. https://ssrn.com/abstract= 3319783
  46. Joana Pais and Ágnes Pintér. 2008. School choice and information: An exper- imental study on matching mechanisms. Games and Economic Behavior 64, 1 (2008), 303-328.
  47. Parag A. Pathak. 2017. What Really Matters in Designing School Choice Mech- anisms. In Advances in Economics and Econometrics: Eleventh World Congress, Bo Honoré, Ariel Pakes, Monika Piazzesi, and Larry Samuelson (Eds.). Econo- metric Society Monographs, Vol. 1. Cambridge University Press, Cambridge, MA, USA, 176-214. https://doi.org/10.1017/9781108227162.006 · DOI
  48. Alex Rees-Jones and Samuel Skowronek. 2018. An experimental investigation of preference misrepresentation in the residency match. Proceedings of the National Academy of Sciences 115, 45 (2018), 11471-11476. https://doi.org/10.1073/pnas. 1803212115 · DOI
  49. Janelle Scott. 2013. School Choice and the Empowerment Imperative. Peabody Journal of Education 88, 1 (2013), 60-73. https://doi.org/10.1080/0161956X.2013. 752635 · DOI
  50. Janelle T. Scott. 2011. Market-Driven Education Reform and the Racial Politics of Advocacy. Peabody Journal of Education 86, 5 (2011), 580-599. https://doi.org/10. 1080/0161956X.2011.616445 · DOI
  51. Andrew D. Selbst, danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* '19). Association for Computing Machinery, New York, NY, USA, 59-68. https://doi.org/10.1145/3287560.3287598 · DOI · Paper page
  52. Lloyd Shapley and Herbert Scarf. 1974. On cores and indivisibility. Journal of Mathematical Economics 1, 1 (1974), 23 -37. https://doi.org/10.1016/0304- 4068(74)90033-0 · DOI
  53. Amy Voida, Lynn Dombrowski, Gillian R. Hayes, and Melissa Mazmanian. 2014. Shared Values/Conficting Logics: Working around e-Government Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 3583-3592. https://doi.org/10.1145/2556288.2556971 · DOI
  54. Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-Sensitive Algorithm Design: Method, Case Study, and Lessons. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 194 (Nov. 2018), 23 pages. https://doi.org/10.1145/ 3274463 · DOI · Paper page

Cited by

Sorted by citations

Authors
Robertson, S., Nguyen, T., & Salehi, N.
Affiliations
University of California, Berkeley (US)
ORCID
Salehi, Niloufar
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-14
DOI
10.1145/3411764.3445748
OpenAlex
W3122939459
Cited by
67
References
8
Citations by year
2026: 4 · 2025: 12 · 2024: 21 · 2023: 13 · 2022: 13 · 2021: 4
FWCI
11.53
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
  • Social Sciences › Education › School Choice and Performance
Keywords
value-sensitive design, algorithmic transparency, educational equity, community, model assumptions, stakeholder engagement, socioeconomic disparities, algorithmic fairness, San Francisco Unified School District, algorithmic governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review