The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study
Abstract
The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants' perceptions of fairness in private while simultaneously allowing for structured participant discussion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deliberation proves effective in facilitating diverse human decision-making tasks, including deliberative politics [13, 52, 119], clinical diagnosis [60, 104, 107], criminal justice [27, 122], among others.”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 2×
“Our work has explored the use of crowdsourcing not from the perspective of contributing new data, but instead on assessing existing model predictors [30].”
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 5×
“Besides reaching a more diverse crowd, our work demonstrates mechanisms to develop what Veale et al. call “ workable social and technical improvements ” to study politically charged settings [55].”
BackgroundInfluentialExplanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388
50 Years of Test (Un)fairness: Lessons for Machine Learning
Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287600 · Cited by 297
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275
A Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Lee, M. K., & Baykal, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998230 · Cited by 182
Fixing the program my computer learned: barriers for end users, challenges for the machine
Kulesza, T., Wong, W.-K., Stumpf, S., Perona, S., White, R., Burnett, M. M., Oberst, I., & Ko, A. J. (2009) · Proceedings of the 14th international conference on Intelligent user interfaces · doi:10.1145/1502650.1502678 · Cited by 83
Grgić-Hlača, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · arXiv · doi:10.48550/arxiv.1802.09548 · Cited by 75
Research works printed but not linked: 38
- 10.1145/3173574.3174156 · DOI · Paper page
- Jonas Auda , Dominik Weber , Alexandra Voit , and Stefan Schneegass . 2018 . Understanding User Preferences Towards Rule-based Notification Deferral. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM , New York, NY, USA, Article LBW584, 6 pages. https://doi.org/10.1145/3170427.3 188688 Jonas Auda, Dominik Weber, Alexandra Voit, and Stefan Schneegass. 2018. Understanding User Preferences Towards Rule-based Notification Deferral. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW584, 6 pages. https://doi.org/10.1145/3170427.3188688 · doi:10.1145/3170427.3
- Philippe Besnard and Anthony Hunter . 2008. Elements of Argumentation . The MIT Press . 298 pages. Philippe Besnard and Anthony Hunter. 2008. Elements of Argumentation .The MIT Press. 298 pages.
- 10.1007/s10618-010-0190-x · DOI · Paper page
- Raj Chetty , Nathaniel Hendren , Patrick Kline , and Emmanuel Saez . 2014. Where is the Land of Opportunity? The Geography of Intergenerational Mobility in the United States. Working Paper 19843 . National Bureau of Economic Research . https://doi.org/10.3386/w 1984 3 Raj Chetty, Nathaniel Hendren, Patrick Kline, and Emmanuel Saez. 2014. Where is the Land of Opportunity? The Geography of Intergenerational Mobility in the United States. Working Paper 19843. National Bureau of Economic Research. https://doi.org/10.3386/w19843 · doi:10.3386/w
- 10.1093/her/cyp046 · DOI · Paper page
- Sarah Desmarais and Jay Singh . 2013. Risk assessment instruments validated and implemented in correctional settings in the United States . Lexington : Council of State Governments ( 2013 ). Sarah Desmarais and Jay Singh. 2013. Risk assessment instruments validated and implemented in correctional settings in the United States. Lexington: Council of State Governments (2013).
- 10.1145/2090236.2090255 · DOI · Paper page
- 10.7709/jnegroeducation.83.3.0371 · doi:10.7709/jnegroeducation.83.3.0371
- 10.2307/2528823 · DOI · Paper page
- Nina Grgiç-Hlavca , Muhammad Bilal Zafar , Krishna P. Gummadi , and Adrian Weller . 2018 . Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , New Orleans, Louisiana, USA . Nina Grgiç-Hlavca, Muhammad Bilal Zafar, Krishna P. Gummadi, and Adrian Weller. 2018. Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA .
- 10.5555/3157382.3157469 · doi:10.5555/3157382.3157469
- Trevor J. Hastie and Robert J . Tibshirani . 1990 . Generalized Additive Models. Monographs on Statistics and Applied Probability, Vol. 43 . Chapman & Hall , London. 352 pages. Trevor J. Hastie and Robert J. Tibshirani. 1990. Generalized Additive Models. Monographs on Statistics and Applied Probability, Vol. 43. Chapman & Hall, London. 352 pages.
- 10.1145/3173574.3173850 · DOI
- 10.2307/256287 · DOI · Paper page
- 10.1145/2702123.2702352 · doi:10.1145/2702123.2702352
- 10.1145/1357054.1357127 · DOI · Paper page
- 10.1145/3064663.3064672 · DOI
- 10.1145/2145204.2145249 · DOI
- M. Lee , L.E. Frank , F. Beute , Y.A.W. de Kort , and W.A. I Jsselsteijn . 2017 . Bots mind the social-technical gap . In Proceedings of 15th European Conference on Computer-Supported Cooperative Work , 28 August - 1 September 2017, Sheffield, United Kingdom (Reports of the European Society for Socially Embedded Technologies). European Society for Socially Embedded Technologies (EUSSET), 35--54. https://doi.org/10. 18420/ecscw2017--14 M. Lee, L.E. Frank, F. Beute, Y.A.W. de Kort, and W.A. IJsselsteijn. 2017. Bots mind the social-technical gap. In Proceedings of 15th European Conference on Computer-Supported Cooperative Work, 28 August - 1 September 2017, Sheffield, United Kingdom (Reports of the European Society for Socially Embedded Technologies). European Society for Socially Embedded Technologies (EUSSET), 35--54. https://doi.org/10.18420/ecscw2017--14 · doi:10.18420/ecscw2017--14
- 10.1145/3025453.3025884 · DOI
- 10.1007/s13347-017-0279-x · DOI · Paper page
- 10.1177/2158244016636433 · DOI · Paper page
- Ritesh Noothigattu , Snehalkumar (Neil) S. Gaikwad , Edmond Awad , Sohan Dsouza , Iyad Rahwan , Pradeep Ravikumar , and Ariel D . Procaccia . 2017 . A Voting-Based System for Ethical Decision Making. CoRR , Vol. abs/ 1709 .06692 (2017). http://arxiv.org/abs/1709.06692 Ritesh Noothigattu, Snehalkumar (Neil) S. Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D. Procaccia. 2017. A Voting-Based System for Ethical Decision Making. CoRR, Vol. abs/1709.06692 (2017). http://arxiv.org/abs/1709.06692 · arXiv
- 10.1145/2998181.2998300 · DOI
- 10.1145/1869086.1869100 · DOI
- 10.1086/268767 · DOI
- 10.1016/j.ijhcs.2009.03.004 · DOI · Paper page
- Edward R. Tufte . 1986. The Visual Display of Quantitative Information . Graphics Press, Cheshire, CT , USA. 200 pages. Edward R. Tufte. 1986. The Visual Display of Quantitative Information .Graphics Press, Cheshire, CT, USA. 200 pages.
- Sherry Turkle . 1995. Life on the Screen: Identity in the Age of the Internet . Simon & Schuster Trade . 352 pages. Sherry Turkle. 1995. Life on the Screen: Identity in the Age of the Internet .Simon & Schuster Trade. 352 pages.
- Melissa A. Valentine , Daniela Retelny , Alexandra To , Negar Rahmati , Tulsee Doshi , and Michael S. Bernstein . 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA, 3523--3537. https://doi.org/10.1145/3025453.3025811 Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3523--3537. https://doi.org/10.1145/3025453.3025811 · DOI · Paper page
- 10.1146/annurev.psych.58.110405.085546 · DOI · Paper page
- Anthony W. Flores , Kristin Bechtel , and Christopher Lowenkamp . 2016. False Positives , False Negatives , and False Analyses : A Rejoinder to "Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks ."., Vol. 80 ( 2016 ). Anthony W. Flores, Kristin Bechtel, and Christopher Lowenkamp. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks."., Vol. 80 (2016).
- 10.1089/109493101753235223 · DOI
- Katherine Y. Williams and Charles A. O'Reilly . 1998 . Demography and Diversity in Organizations: A Review of 40 Years of Research . Research in Organizational Behavior , Vol. 20 (1998), 77 -- 140 . Katherine Y. Williams and Charles A. O'Reilly. 1998. Demography and Diversity in Organizations: A Review of 40 Years of Research. Research in Organizational Behavior, Vol. 20 (1998), 77--140.
- S.N Wood . 2017. Generalized Additive Models: An Introduction with R 2 ed.) . Chapman and Hall/CRC. S.N Wood. 2017. Generalized Additive Models: An Introduction with R 2 ed.). Chapman and Hall/CRC.
- 10.1145/2531602.2531604 · DOI
- Sharon Zhou , Melissa Valentine , and Michael S. Bernstein . 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18) . ACM, New York, NY, USA, Article 108, 13 pages. https://doi.org/10.1145/3173574.3173682 Sharon Zhou, Melissa Valentine, and Michael S. Bernstein. 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 108, 13 pages. https://doi.org/10.1145/3173574.3173682 · DOI
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“asks and advise human decision makers on higher stake decisions, such as medical diagnosis [11, 31], wildlife protection [19], improving housing for homeless youth [12, 29], and recidivism prediction [23, 25, 60]. While there are many motivations for creating these human-AI teams, a primary one is increasing the effec…”
BackgroundFairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221
Metadata
- Authors
- van Berkel, N., Goncalves, J., Hettiachchi, D., Wijenayake, S., Kelly, R. M., & Kostakos, V.
- Affiliations
- The University of Melbourne, Melbourne, Australia
- The University of Melbourne, Melbourne, VIC, Australia
- University of Melbourne, Melbourne, VIC, Australia
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2019-11-07
- Volume
- 3
- Issue
- CSCW
- Pages
- 1-21
- ISSN
- 2573-0142
- Cited by
- 60
- References
- 22
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
