Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 72

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation

Zhang, Angie, Walker, Olympia, Nguyen, Kaci, Dai, Jiajun, Chen, Anqing, Lee, Min Kyung · 2023 · Proceedings of the ACM on Human-Computer Interaction

Abstract

Research exploring how to support decision-making has often used machine learning to automate or assist human decisions. We take an alternative approach for improving decision-making, using machine learning to help stakeholders surface ways to improve and make fairer decision-making processes. We created "Deliberating with AI", a web tool that enables people to create and evaluate ML models in order to examine strengths and shortcomings of past decision-making and deliberate on how to improve future decisions. We apply this tool to a context of people selection, having stakeholders---decision makers (faculty) and decision subjects (students)---use the tool to improve graduate school admission decisions. Through our case study, we demonstrate how the stakeholders used the web tool to create ML models that they used as boundary objects to deliberate over organization decision-making practices. We share insights from our study to inform future research on stakeholder-centered participatory AI design and technology for organizational decision-making.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3579601

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 72

  1. 10.1109/ICCIDS.2019.8862140 · DOI · Paper page
  2. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica, Vol. 23 (2016), 77--91.
  3. Cecilia Aragon, Shion Guha, Marina Kogan, Michael Muller, and Gina Neff. 2022. Human-Centered Data Science: An Introduction. MIT Press.
  4. 10.1093/qje/qjy012 · DOI · Paper page
  5. 10.1038/s41598-020-73246-2 · DOI
  6. 10.1080/00221546.2018.1442633 · DOI
  7. 10.1145/1978942.1979275 · DOI · Paper page
  8. Marianne Bertrand and Sendhil Mullainathan. 2004. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review, Vol. 94, 4 (2004), 991--1013.
  9. Sarah Bird, Miro Dud'ik, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020. Fairlearn: A toolkit for assessing and improving fairness in AI. Microsoft, Tech. Rep. MSR-TR-2020--32 (2020).
  10. 10.1145/3290605.3300271 · DOI · Paper page
  11. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91.
  12. Stephanie Burkhalter, John Gastil, and Todd Kelshaw. 2002. A conceptual definition and theoretical model of public deliberation in small face-to-face groups. Communication theory, Vol. 12, 4 (2002), 398--422.
  13. 10.1145/3411763.3443435 · DOI
  14. 10.1016/j.ijhcs.2021.102607 · DOI
  15. 10.1145/3411764.3445308 · DOI
  16. Bo Cowgill and Catherine Tucker. 2017. Algorithmic bias: A counterfactual perspective. NSF Trustworthy Algorithms (2017).
  17. Robyn M Dawes, David Faust, and Paul E Meehl. 1989. Clinical versus actuarial judgment. Science, Vol. 243, 4899 (1989), 1668--1674.
  18. Maria De-Arteaga, Alexandra Chouldechova, and Artur Dubrawski. 2022. Doubting AI Predictions: Influence-Driven Second Opinion Recommendation. arXiv preprint arXiv:2205.00072 (2022). · arXiv
  19. Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2021. Stakeholder Participation in AI: Beyond" Add Diverse Stakeholders and Stir". arXiv preprint arXiv:2111.01122 (2021). · arXiv
  20. 10.1075/idj.23.1.03dig · DOI
  21. Catherine D'ignazio and Lauren F Klein. 2020. Data feminism. MIT press.
  22. 10.1609/hcomp.v4i1.13270 · DOI
  23. Selen A Ercan, Carolyn M Hendriks, and John S Dryzek. 2019. Public deliberation in an era of communicative plenty. Policy & politics, Vol. 47, 1 (2019), 19--36.
  24. 10.1145/3313831.3376293 · DOI
  25. 10.1145/358916.359001 · DOI
  26. James S Fishkin. 2002. Deliberative democracy. The Blackwell guide to social and political philosophy (2002), 221--238.
  27. Archon Fung. 2007. Minipublics: Deliberative designs and their consequences. In Deliberation, participation and democracy. Springer, 159--183.
  28. Claudia Goldin and Cecilia Rouse. 2000. Orchestrating impartiality: The impact of" blind" auditions on female musicians. American economic review, Vol. 90, 4 (2000), 715--741.
  29. 10.1111/j.0032-3217.2003.00450.x · DOI · Paper page
  30. Nina Grgi?-Hlavc a, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2018. Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
  31. William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. 2000. Clinical versus mechanical prediction: a meta-analysis. Psychological assessment, Vol. 12, 1 (2000), 19.
  32. 10.1145/3359318 · DOI · Paper page
  33. 10.18608/jla.2019.62.3 · DOI · Paper page
  34. 10.1145/3290605.3300291 · DOI
  35. 10.1145/985921.986201 · DOI
  36. Julia D Kent and Maureen Terese McCarthy. 2016. Holistic review in graduate admissions. Council of Graduate Schools (2016).
  37. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics, Vol. 133, 1 (2018), 237--293.
  38. Margaret Kramer. 2019. A timeline of key Supreme Court cases on affirmative action. New York Times, Vol. 30 (2019).
  39. 10.1145/2145204.2145249 · DOI · Paper page
  40. 10.1145/2675133.2675147 · DOI · Paper page
  41. 10.1007/s10606-007-9044-5 · DOI · Paper page
  42. 10.1145/3359284 · DOI · Paper page
  43. 10.1145/2750858.2807552 · DOI
  44. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019b. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1--35.
  45. 10.1145/2030112.2030166 · DOI · Paper page
  46. Caitlin Lustig, Artie Konrad, and Jed R Brubaker. 2022. Designing for the Bittersweet: Improving Sensitive Experiences with Recommender Systems. In CHI Conference on Human Factors in Computing Systems. 1--18.
  47. David Madras, Toniann Pitassi, and Richard Zemel. 2017. Predict responsibly: improving fairness and accuracy by learning to defer. arXiv preprint arXiv:1711.06664 (2017). · arXiv
  48. Laura Matrajt, Julia Eaton, Tiffany Leung, and Elizabeth R Brown. 2021. Vaccine optimization for COVID-19: Who to vaccinate first? Science Advances, Vol. 7, 6 (2021), eabf1374.
  49. 10.1145/3457607 · DOI · Paper page
  50. Ronald A Metoyer, Tya S Chuanromanee, Gina M Girgis, Qiyu Zhi, and Eleanor C Kinyon. 2020. Supporting Storytelling With Evidence in Holistic Review Processes: A Participatory Design Approach. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW1 (2020), 1--24.
  51. 10.1145/3397481.3450658 · DOI
  52. Amy Hawn Nelson, Della Jenkins, Sharon Zanti, Matthew F Katz, Emily Berkowitz, TC Burnett, and Dennis P Culhane. 2020. A toolkit for centering racial equity throughout data integration. Actionable Intelligence for Social Policy, University of Pennsylvania.
  53. Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE Publications, inc.
  54. Julie R Posselt. 2016. Inside graduate admissions: Merit, diversity, and faculty gatekeeping. Harvard University Press.
  55. 10.1145/3359178 · DOI · Paper page
  56. 10.1145/3274423 · DOI
  57. Donald A Schön. 2017. The reflective practitioner: How professionals think in action. Routledge.
  58. 10.1145/3442188.3445971 · DOI
  59. 10.1145/3415224 · DOI
  60. 10.1145/3531146.3533110 · DOI
  61. Susan Leigh Star. 1989. The structure of ill-structured solutions: Boundary objects and heterogeneous distributed problem solving. In Distributed artificial intelligence. Elsevier, 37--54.
  62. Poorna Talkad Sukumar and Ronald Metoyer. 2018. A visualization approach to addressing reviewer bias in holistic college admissions. In Cognitive Biases in Visualizations. Springer, 161--175.
  63. Poorna Talkad Sukumar, Ronald Metoyer, and Shuai He. 2018. Making a pecan pie: Understanding and supporting the holistic review process in admissions. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 1--22.
  64. Halil Toros and Daniel Flaming. 2018. Prioritizing homeless assistance using predictive algorithms: an evidence-based approach. Cityscape, Vol. 20, 1 (2018), 117--146.
  65. Grutter v Bollinger. 2014. 539 US 306 (2003). Legal Information Institute, Cornell University Law School. https://www. law. cornell. edu/supct/html/02--241. ZO. html (2014).
  66. 10.1145/3359130 · DOI · Paper page
  67. 10.25300/MISQ/2021/16559 · DOI · Paper page
  68. 10.1609/aimag.v35i1.2504 · DOI · Paper page
  69. James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics, Vol. 26, 1 (2019), 56--65.
  70. Sarah Williams, Erica Deahl, Laurie Rubel, and Vivian Lim. 2014. City Digits: Local Lotto: Developing youth data literacy by investigating the lottery. Journal of Digital Media Literacy (2014).
  71. Zining Ye, Xinran Yuan, Shaurya Gaur, Aaron Halfaker, Jodi Forlizzi, and Haiyi Zhu. 2021. Wikipedia ORES Explorer: Visualizing Trade-offs For Designing Applications With Machine Learning API. In Designing Interactive Systems Conference 2021. 1554--1565.
  72. 10.1145/3357236.3395528 · DOI

Cited by

See all 72

Metadata

Authors
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K.
Affiliations
The University of Texas at Austin, Austin, TX, USA
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2023-04-14
Volume
7
Issue
CSCW1
Pages
1-32
ISSN
2573-0142
Cited by
72
References
34
Language
en
Funders
NSF (1952085, 1939606, 2125858)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation. https://app.cochat.ai/papers/10.1145/3579601
The paper
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023). Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3579601

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.