CoChat
Sign Up
Cited by 190

A Qualitative Exploration of Perceptions of Algorithmic Fairness

Woodruff, Allison, Fox, Sarah E., Rousso-Schindler, Steven, Warshaw, Jeffrey·2018·Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3173574.3174230
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3173574.3174230

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Past research in HCI and social computing has developed and studied a host of fairness, explainability, and other toolkits that inform AI evaluations.

    “Past research in human-computer interaction (HCI) and social computing has developed and studied a host of fairness, explainability, and other toolkits that inform such evaluations [5, 15, 39, 54, 69, 70, 72, 80, 82, 85, 104, 119, 125, 126, 132].”

    From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [5]

    In “Workshop as Method” (page unavailable), the paper describes using a workshop as a research instrument to study perceptions of algorithmic fairness and inform future technology and policy. That supports the sentence’s broader point about HCI methods informing fairness work, but our copy does not describe this study as developing or studying a fairness or explainability toolkit for AI evaluations. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. ACM US Public Policy Council. 2017. Statement on Algorithmic Transparency and Accountability. Retrieved September 16, 2017 from https://www.acm.org/binaries/content/assets/public- policy/2017_usacm_statement_algorithms.pdf
  2. Mariam Asad, Sarah Fox, and Christopher A. Le Dantec. 2014. Speculative Activist Technologies. Proceedings iConference 2014. https://doi.org/10.9776/14074 · DOI
  3. Paul Baker and Amanda Potts. 2013. 'Why do white people have thin lips?' Google and the perpetuation of stereotypes via auto-complete search forms. Critical Discourse Studies 10, 2: 187-204. https://doi.org/10.1080/17405904.2012.744320 · DOI · Paper page
  4. David Beer. 2009. Power through the algorithm? Participatory web cultures and the technological unconscious. New Media & Society 11, 6: 985-1002. https://doi.org/10.1177/1461444809336551 · DOI
  5. Hugh Beyer and Karen Holtzblatt. 1998. Contextual Design: Defining Customer-centered Systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  6. danah boyd and Kate Crawford. 2012. Critical Questions for Big Data. Information, Communication & Society 15, 5: 662-679. https://doi.org/10.1080/1369118X.2012.678878 · DOI · Paper page
  7. danah boyd, Karen Levy, and Alice Marwick. 2014. The Networked Nature of Algorithmic Discrimination. In Data and Discrimination: Collected Essays, Seeta Peña Gangadharan, Virginia Eubanks and Solon Barocas (eds.). Open Technology Institute, New America Foundation, Washington, D.C., 53-57.
  8. Engin Bozdag. 2013. Bias in algorithmic filtering and personalization. Ethics and Information Technology 15, 3: 209-227. https://doi.org/10.1007/s10676-013-9321- 6 · DOI · Paper page
  9. Danielle Keats Citron and Frank Pasquale. 2014. The Scored Society: Due Process for Automated Predictions. Washington Law Review 89, 1.
  10. Ashley Colley, Jacob Thebault-Spieker, Allen Yilun Lin, Donald Degraen, Benjamin Fischman, Jonna Häkkilä, Kate Kuehl, Valentina Nisi, Nuno Jardim Nunes, Nina Wenig, Dirk Wenig, Brent Hecht, and Johannes Schöning. 2017. The Geography of PokéMon GO: Beneficial and Problematic Effects on Places and Movement. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1179-1192. https://doi.org/10.1145/3025453.3025495 · DOI
  11. Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2015. Automated Experiments on Ad Privacy Settings. In Proceedings on Privacy Enhancing Technologies (PETS 2015), 92-112. https://doi.org/10.1515/popets-2015-0007 · DOI · Paper page
  12. Michael A. DeVito, Jeremy Birnholtz, and Jeffery T. Hancock. Platforms, People, and Perception: Using Affordances to Understand Self-Presentation on Social Media. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '17), 740-754.
  13. Nicholas Diakopoulos. 2015. Algorithmic Accountability. Digital Journalism 3, 3: 398-415. https://doi.org/10.1080/21670811.2014.976411 · DOI · Paper page
  14. Carl DiSalvo, Thomas Lodato, Laura Fries, Beth Schechter, and Thomas Barnwell. 2011. The collective articulation of issues as design practice. CoDesign 7, 3-4: 185-197. https://doi.org/10.1080/15710882.2011.630475 · DOI
  15. Carl DiSalvo, Illah Nourbakhsh, David Holstius, Ayça Akin, and Marti Louw. 2008. The Neighborhood Networks Project: A Case Study of Critical Engagement and Creative Expression Through Participatory Design. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (PDC '08), 41-50.
  16. Carl DiSalvo, Phoebe Sengers, and Hrönn Brynjarsdóttir. 2010. Mapping the Landscape of Sustainable HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 1975-1984. https://doi.org/10.1145/1753326.1753625 · DOI
  17. Paul Dourish. 2006. Implications for Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), 541-550. https://doi.org/10.1145/1124772.1124855 · DOI · Paper page
  18. Benjamin Edelman, Michael Luca, and Daniel Svirsky. 2017. Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment. American Economic Journal: Applied Economics 9, 2: 1-22.
  19. Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. "I Always Assumed That I Wasn't Really That Close to [Her]": Reasoning About Invisible Algorithms in News Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 153-162. https://doi.org/10.1145/2702123.2702556 · DOI
  20. Daniel C. Esty and Andrew Winston. 2006. Green to Gold: How Smart Companies Use Environmental Strategy to Innovate, Create Value, and Build Competitive Advantage. Yale University Press.
  21. Shaun L. Gabbidon. 2003. Racial Profiling by Store Clerks and Security Personnel in Retail Establishments: An Exploration of "Shopping While Black." Journal of Contemporary Criminal Justice 19, 3: 345-364. https://doi.org/10.1177/1043986203254531 · DOI
  22. Tarleton Gillespie. 2014. The relevance of algorithms. In Media Technologies: Essays on Communication, Materiality, and Society -University Press Scholarship, Tarleton Gillespie, Pablo Boczkowski and Kirsten Foot (eds.). MIT Press.
  23. Anthony G. Greenwald, Mahzarin R. Banaji, and Brian A. Nosek. 2015. Statistically small effects of the Implicit Association Test can have societally large effects. Journal of Personality and Social Psychology 108, 4: 553-561. https://doi.org/10.1037/pspa0000016 · DOI
  24. Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, and Alexander van Esbroeck. 2016. Monotonic Calibrated Interpolated Look-Up Tables. Journal of Machine Learning Research 17, 109: 1-47.
  25. Bernard E. Harcourt. 2007. Against prediction: Profiling, policing, and punishing in an actuarial age. University of Chicago Press.
  26. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems (NIPS 2016), 3315-3323.
  27. Shin Inouye. 2016. Advocates Applaud Google's Ban on Payday Loan Advertisements. The Leadership Conference on Civil and Human Rights. Retrieved September 15, 2017 from http://civilrights.org/advocates-applaud-googles-ban- on-payday-loan-advertisements/
  28. Lucas D. Introna and Helen Nissenbaum. 2000. Shaping the Web: Why the Politics of Search Engines Matters. The Information Society 16, 3: 169-185. https://doi.org/10.1080/01972240050133634 · DOI
  29. Isaac Johnson, Connor McMahon, Johannes Schöning, and Brent Hecht. 2017. The Effect of Population and "Structural" Biases on Social Media-based Algorithms: A Case Study in Geolocation Inference Across the Urban-Rural Spectrum. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1167-1178. https://doi.org/10.1145/3025453.3026015 · DOI
  30. Robert Jungk, Norbert Müllert, and Institute for Social Inventions. 1987. Future workshops: how to create desirable futures. Institute for Social Inventions, London.
  31. Jonathan Lazar, Julio Abascal, Simone Barbosa, Jeremy Barksdale, Batya Friedman, Jens Grossklags, Jan Gulliksen, Jeff Johnson, Tom McEwan, Loïc Martínez-Normand, Wibke Michalk, Janice Tsai, Gerrit van der Veer, Hans von Axelson, Ake Walldius, Gill Whitney, Marco Winckler, Volker Wulf, Elizabeth F. Churchill, Lorrie Cranor, Janet Davis, Alan Hedge, Harry Hochheiser, Juan Pablo Hourcade, Clayton Lewis, Lisa Nathan, Fabio Paterno, Blake Reid, Whitney Quesenbery, Ted Selker, and Brian Wentz. 2016. Human-Computer Interaction and International Public Policymaking: A Framework for Understanding and Taking Future Actions. Foundations and Trends in Human-Computer Interaction 9, 2: 69-149. https://doi.org/10.1561/1100000062 · DOI
  32. Min Kyung Lee and Su Baykal. 2017. Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '17), 1035-1048.
  33. Min Kyung Lee, Ji Tae Kim, and Leah Lizarondo. 2017. A Human-Centered Approach to Algorithmic Services: Considerations for Fair and Motivating Smart Community Service Management that Allocates Donations to Non-Profit Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 3365-3376.
  34. Richard J. Lundman and Robert L. Kaufman. 2003. Driving While Black: Effects of Race, Ethnicity, and Gender on Citizen Self-Reports of Traffic Stops and Police Actions. Criminology 41, 1: 195-220. https://doi.org/10.1111/j.1745-9125.2003.tb00986.x · DOI
  35. Caitlin Lustig, Katie Pine, Bonnie Nardi, Lilly Irani, Min Kyung Lee, Dawn Nafus, and Christian Sandvig. 2016. Algorithmic Authority: The Ethics, Politics, and Economics of Algorithms That Interpret, Decide, and Manage. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16), 1057-1062. https://doi.org/10.1145/2851581.2886426 · DOI
  36. Amanda Menking and Ingrid Erickson. 2015. The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's Largest Online Encyclopedia. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 207- 210. https://doi.org/10.1145/2702123.2702514 · DOI · Paper page
  37. Carey Nadeau and Amy K. Glasmeier. 2016. Minimum Wage: Can an Individual or a Family Live on It? Retrieved September 15, 2017 from http://livingwage.mit.edu/articles/15-minimum-wage- can-an-individual-or-a-family-live-on-it 62. Safiya Umoja Noble. 2014. Teaching Trayvon: Race, Media, and the Politics of Spectacle. The Black Scholar 44, 1: 12-29. https://doi.org/10.5816/blackscholar.44.1.0012 · DOI
  38. Jahna Otterbacher, Jo Bates, and Paul Clough. 2017. Competent Men and Warm Women: Gender Stereotypes and Backlash in Image Search Results. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 6620-6631. https://doi.org/10.1145/3025453.3025727 · DOI
  39. Sarah Pink. 2014. Doing Visual Ethnography. Sage Publications.
  40. Angelisa Plane, Elissa Redmiles, Michelle Mazurek, and Michael Tschantz. 2017. Exploring User Perceptions of Discrimination in Online Targeted Advertising. In Proceedings of the 2017 USENIX Security Symposium.
  41. Matt Ratto. 2011. Critical Making: Conceptual and Material Studies in Technology and Social Life. The Information Society 27, 4: 252-260. https://doi.org/10.1080/01972243.2011.583819 · DOI
  42. Noopur Raval and Paul Dourish. 2016. Standing Out from the Crowd: Emotional Labor, Body Labor, and Temporal Labor in Ridesharing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 97-107. https://doi.org/10.1145/2818048.2820026 · DOI
  43. Daniela K. Rosner, Saba Kawas, Wenqi Li, Nicole Tilly, and Yi-Chen Sung. 2016. Out of Time, Out of Place: Reflections on Design Workshops as a Research Method. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 1131-1141. https://doi.org/10.1145/2818048.2820021 · DOI
  44. Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. CoDesign 4, 1: 5-18. https://doi.org/10.1080/15710880701875068 · DOI · Paper page
  45. Sassafras Tech Collective. 2016. Icebreaker. In Exploring Social Justice, Design, and HCI Workshop at CHI 2016.
  46. Kiley Sobel, Katie O'Leary, and Julie A. Kientz. 2015. Maximizing Children's Opportunities with Inclusive Play: Considerations for Interactive Technology Design. In Proceedings of the 14th International Conference on Interaction Design and Children (IDC '15), 39-48. https://doi.org/10.1145/2771839.2771844 · DOI
  47. Derald Wing Sue. 2010. Microaggressions in Everyday Life: Race, Gender, and Sexual Orientation. Wiley.
  48. Vanessa Thomas, Christian Remy, Mike Hazas, and Oliver Bates. 2017. HCI and Environmental Public Policy: Opportunities for Engagement. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 6986-6992. https://doi.org/10.1145/3025453.3025579 · DOI
  49. Blase Ur, Pedro Giovanni Leon, Lorrie Faith Cranor, Richard Shay, and Yang Wang. 2012. Smart, Useful, Scary, Creepy: Perceptions of Online Behavioral Advertising. In Proceedings of the Eighth Symposium on Usable Privacy and Security (SOUPS '12), 4:1- 4:15. https://doi.org/10.1145/2335356.2335362 · DOI
  50. Jeff Warshaw, Nina Taft, and Allison Woodruff. 2016. Intuitions, analytics, and killing ants: Inference literacy of high school-educated adults in the US. In Proceedings of the Twelfth Symposium on Usable Privacy and Security (SOUPS '16).

Cited by

Sorted by citations

Authors
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J.
Affiliations
  • Google, Inc., Mountain View, CA, USA
  • Google, Inc. & University of Washington, Seattle, WA, USA
  • California State University Long Beach, Long Beach, CA, USA
Venue
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2018-04-21
Pages
1-14
DOI
10.1145/3173574.3174230
Cited by
190
References
11
Licence
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review