Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 60

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study

van Berkel, Niels, Goncalves, Jorge, Hettiachchi, Danula, Wijenayake, Senuri, Kelly, Ryan M., Kostakos, Vassilis·2019·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3359130
Open this paper in CoChat — free account

Abstract

The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants' perceptions of fairness in private while simultaneously allowing for structured participant discussion.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3359130

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 22 references

Research works printed but not linked: 38

  1. 10.1145/3173574.3174156 · DOI · Paper page
  2. Jonas Auda , Dominik Weber , Alexandra Voit , and Stefan Schneegass . 2018 . Understanding User Preferences Towards Rule-based Notification Deferral. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM , New York, NY, USA, Article LBW584, 6 pages. https://doi.org/10.1145/3170427.3 188688 Jonas Auda, Dominik Weber, Alexandra Voit, and Stefan Schneegass. 2018. Understanding User Preferences Towards Rule-based Notification Deferral. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW584, 6 pages. https://doi.org/10.1145/3170427.3188688 · doi:10.1145/3170427.3
  3. Philippe Besnard and Anthony Hunter . 2008. Elements of Argumentation . The MIT Press . 298 pages. Philippe Besnard and Anthony Hunter. 2008. Elements of Argumentation .The MIT Press. 298 pages.
  4. 10.1007/s10618-010-0190-x · DOI · Paper page
  5. Raj Chetty , Nathaniel Hendren , Patrick Kline , and Emmanuel Saez . 2014. Where is the Land of Opportunity? The Geography of Intergenerational Mobility in the United States. Working Paper 19843 . National Bureau of Economic Research . https://doi.org/10.3386/w 1984 3 Raj Chetty, Nathaniel Hendren, Patrick Kline, and Emmanuel Saez. 2014. Where is the Land of Opportunity? The Geography of Intergenerational Mobility in the United States. Working Paper 19843. National Bureau of Economic Research. https://doi.org/10.3386/w19843 · doi:10.3386/w
  6. 10.1093/her/cyp046 · DOI · Paper page
  7. Sarah Desmarais and Jay Singh . 2013. Risk assessment instruments validated and implemented in correctional settings in the United States . Lexington : Council of State Governments ( 2013 ). Sarah Desmarais and Jay Singh. 2013. Risk assessment instruments validated and implemented in correctional settings in the United States. Lexington: Council of State Governments (2013).
  8. 10.1145/2090236.2090255 · DOI · Paper page
  9. 10.7709/jnegroeducation.83.3.0371 · doi:10.7709/jnegroeducation.83.3.0371
  10. 10.2307/2528823 · DOI · Paper page
  11. Nina Grgiç-Hlavca , Muhammad Bilal Zafar , Krishna P. Gummadi , and Adrian Weller . 2018 . Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , New Orleans, Louisiana, USA . Nina Grgiç-Hlavca, Muhammad Bilal Zafar, Krishna P. Gummadi, and Adrian Weller. 2018. Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA .
  12. 10.5555/3157382.3157469 · doi:10.5555/3157382.3157469
  13. Trevor J. Hastie and Robert J . Tibshirani . 1990 . Generalized Additive Models. Monographs on Statistics and Applied Probability, Vol. 43 . Chapman & Hall , London. 352 pages. Trevor J. Hastie and Robert J. Tibshirani. 1990. Generalized Additive Models. Monographs on Statistics and Applied Probability, Vol. 43. Chapman & Hall, London. 352 pages.
  14. 10.1145/3173574.3173850 · DOI
  15. 10.2307/256287 · DOI · Paper page
  16. 10.1145/2702123.2702352 · doi:10.1145/2702123.2702352
  17. 10.1145/1357054.1357127 · DOI · Paper page
  18. 10.1145/3064663.3064672 · DOI
  19. 10.1145/2145204.2145249 · DOI
  20. M. Lee , L.E. Frank , F. Beute , Y.A.W. de Kort , and W.A. I Jsselsteijn . 2017 . Bots mind the social-technical gap . In Proceedings of 15th European Conference on Computer-Supported Cooperative Work , 28 August - 1 September 2017, Sheffield, United Kingdom (Reports of the European Society for Socially Embedded Technologies). European Society for Socially Embedded Technologies (EUSSET), 35--54. https://doi.org/10. 18420/ecscw2017--14 M. Lee, L.E. Frank, F. Beute, Y.A.W. de Kort, and W.A. IJsselsteijn. 2017. Bots mind the social-technical gap. In Proceedings of 15th European Conference on Computer-Supported Cooperative Work, 28 August - 1 September 2017, Sheffield, United Kingdom (Reports of the European Society for Socially Embedded Technologies). European Society for Socially Embedded Technologies (EUSSET), 35--54. https://doi.org/10.18420/ecscw2017--14 · doi:10.18420/ecscw2017--14
  21. 10.1145/3025453.3025884 · DOI
  22. 10.1007/s13347-017-0279-x · DOI · Paper page
  23. 10.1177/2158244016636433 · DOI · Paper page
  24. Ritesh Noothigattu , Snehalkumar (Neil) S. Gaikwad , Edmond Awad , Sohan Dsouza , Iyad Rahwan , Pradeep Ravikumar , and Ariel D . Procaccia . 2017 . A Voting-Based System for Ethical Decision Making. CoRR , Vol. abs/ 1709 .06692 (2017). http://arxiv.org/abs/1709.06692 Ritesh Noothigattu, Snehalkumar (Neil) S. Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D. Procaccia. 2017. A Voting-Based System for Ethical Decision Making. CoRR, Vol. abs/1709.06692 (2017). http://arxiv.org/abs/1709.06692 · arXiv
  25. 10.1145/2998181.2998300 · DOI
  26. 10.1145/1869086.1869100 · DOI
  27. 10.1086/268767 · DOI
  28. 10.1016/j.ijhcs.2009.03.004 · DOI · Paper page
  29. Edward R. Tufte . 1986. The Visual Display of Quantitative Information . Graphics Press, Cheshire, CT , USA. 200 pages. Edward R. Tufte. 1986. The Visual Display of Quantitative Information .Graphics Press, Cheshire, CT, USA. 200 pages.
  30. Sherry Turkle . 1995. Life on the Screen: Identity in the Age of the Internet . Simon & Schuster Trade . 352 pages. Sherry Turkle. 1995. Life on the Screen: Identity in the Age of the Internet .Simon & Schuster Trade. 352 pages.
  31. Melissa A. Valentine , Daniela Retelny , Alexandra To , Negar Rahmati , Tulsee Doshi , and Michael S. Bernstein . 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA, 3523--3537. https://doi.org/10.1145/3025453.3025811 Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3523--3537. https://doi.org/10.1145/3025453.3025811 · DOI · Paper page
  32. 10.1146/annurev.psych.58.110405.085546 · DOI · Paper page
  33. Anthony W. Flores , Kristin Bechtel , and Christopher Lowenkamp . 2016. False Positives , False Negatives , and False Analyses : A Rejoinder to "Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks ."., Vol. 80 ( 2016 ). Anthony W. Flores, Kristin Bechtel, and Christopher Lowenkamp. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks."., Vol. 80 (2016).
  34. 10.1089/109493101753235223 · DOI
  35. Katherine Y. Williams and Charles A. O'Reilly . 1998 . Demography and Diversity in Organizations: A Review of 40 Years of Research . Research in Organizational Behavior , Vol. 20 (1998), 77 -- 140 . Katherine Y. Williams and Charles A. O'Reilly. 1998. Demography and Diversity in Organizations: A Review of 40 Years of Research. Research in Organizational Behavior, Vol. 20 (1998), 77--140.
  36. S.N Wood . 2017. Generalized Additive Models: An Introduction with R 2 ed.) . Chapman and Hall/CRC. S.N Wood. 2017. Generalized Additive Models: An Introduction with R 2 ed.). Chapman and Hall/CRC.
  37. 10.1145/2531602.2531604 · DOI
  38. Sharon Zhou , Melissa Valentine , and Michael S. Bernstein . 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18) . ACM, New York, NY, USA, Article 108, 13 pages. https://doi.org/10.1145/3173574.3173682 Sharon Zhou, Melissa Valentine, and Michael S. Bernstein. 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 108, 13 pages. https://doi.org/10.1145/3173574.3173682 · DOI

Cited by

Metadata

Authors
van Berkel, N., Goncalves, J., Hettiachchi, D., Wijenayake, S., Kelly, R. M., & Kostakos, V.
Affiliations
  • The University of Melbourne, Melbourne, Australia
  • The University of Melbourne, Melbourne, VIC, Australia
  • University of Melbourne, Melbourne, VIC, Australia
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-07
Volume
3
Issue
CSCW
Pages
1-21
ISSN
2573-0142
DOI
10.1145/3359130
Cited by
60
References
22
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.