The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Everyday Algorithm Auditing: Understanding the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors
Open accessAbstract
A growing body of literature has proposed formal approaches to audit algorithmic systems for biased and harmful behaviors. While formal auditing approaches have been greatly impactful, they often suffer major blindspots, with critical issues surfacing only in the context of everyday use once systems are deployed. Recent years have seen many cases in which everyday users of algorithmic systems detect and raise awareness about harmful behaviors that they encounter in the course of their everyday interactions with these systems. However, to date little academic attention has been granted to these bottom-up, user-driven auditing processes. In this paper, we propose and explore the concept of everyday algorithm auditing, a process in which users detect, understand, and interrogate problematic machine behaviors via their day-to-day interactions with algorithmic systems. We argue that everyday users are powerful in surfacing problematic machine behaviors that may elude detection via more centrally-organized forms of auditing, regardless of users' knowledge about the underlying algorithms. We analyze several real-world cases of everyday algorithm auditing, drawing lessons from these cases for the design of future platforms and tools that facilitate such auditing behaviors. Finally, we discuss work that lies ahead, toward bridging the gaps between formal auditing approaches and the organic auditing behaviors that emerge in everyday use of algorithmic systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, by providing a basis on which technology providers can be held accountable, audits incentivise investments in adequate risk management (Shen et al., 2021).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Shen et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While there is a growing interest in auditing algorithmic systems [125,126,128,138,154], this feld generally lacks defned standards and records could be incomplete or not relating to information needed for accountability purposes [15,127].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [138]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, a recent line of HCI research has explored the idea of”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [138]
The cited source says what the claim attributes to it.
In the citing paper
“Recent HCI work on user auditing [132], such as the 'WeAudit' tool [126], exemplifies this shift and demonstrates the possibility of designing for a more participatory AI audit process [2, 52, 132].”
From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Public Sphere: A Contribution to the Critique of Actually Existing Democracy
Fraser, N. (1990) · Social Text · doi:10.2307/466240 · Cited by 5,147
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 1×
“In response to these concerns, scholars have proposed a number of approaches to audit algorithmic systems for biased, discriminatory, or otherwise harmful behaviors 1 — both internally, within the organizations responsible for developing and maintaining these systems, and externally [73, 76].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 2×
“Other harmful behaviors may only emerge when a system is used in unanticipated ways or in unanticipated contexts, perhaps due to changing norms and practices surrounding the use of a given algorithmic system over time [18, 82].”
BackgroundMeme-tracking and the dynamics of the news cycle
Leskovec, J., Backstrom, L., & Kleinberg, J. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557077 · Cited by 1,530 · cited 1×
“This is a particularly appealing possibility given that information in memes tends to propagate from mainstream news to social media [58].”
BackgroundPredicting protein structures with a multiplayer online game
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., & players, F. (2010) · Nature · doi:10.1038/nature09304 · Cited by 1,368 · cited 1×
“Past work has explored collective sensemaking across a range of contexts, including citizen science [17], image labelling [96], knowledge mapping and curation (e.”
BackgroundFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Algorithms as culture: Some tactics for the ethnography of algorithmic systems
Seaver, N. (2017) · Big Data & Society · doi:10.1177/2053951717738104 · Cited by 1,010 · cited 3×
“These factors make detecting certain harmful algorithmic behaviors difficult for auditors unless they are embedded within particular real-world social or cultural contexts [18, 49, 80, 101].”
BackgroundMemes in a Digital World: Reconciling with a Conceptual Troublemaker
Shifman, L. (2013) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12013 · Cited by 898
Research works printed but not linked: 27
- Todd Bishop. 2015. Amazon changes its key formula for calculating product ratings and displaying reviews. GeekWire (June 2015). https://www.geekwire.com/2015/amazon-changes-its-influential-formula-for-calculating-productratings
- Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050 (2020). · arXiv
- Rucksack Brian. 2019. Major changes to review scoring method on Booking com. https://hostelmanagement.com/forums/major-changes-review-scoring-method-booking-com.html
- Joy Buolamwini. 2016. How I'm fighting bias in algorithms. November, TEDx Beacon Street)[Video File] (2016).
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency PMLR. 77--91.
- Kate Crawford and Trevor Paglen. 2021. Excavating AI: The politics of images in machine learning training sets. AI & Society (2021), 1--12.
- Richard Dawkins. 2016. The selfish gene. Oxford University Press.
- Michel De Certeau. 1984. The practice of everyday life, trans Steven Rendall. Berkeley: University of California Press.
- Nicholas Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. The Tow Center for Digital Journalism (2014).
- Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a "right to explanation". In ICML Workshop on Human Interpretability in Machine Learning.
- Jürgen Habermas. 1989. The structural transformation of the public sphere: An inquiry into a category of bourgeois society. MIT press.
- White House. 2016. Big Data: A Report on Algorithmic Systems. Opportunity, and Civil Rights (2016).
- Inkoo Kang. 2013. Businesses: ?Yelp is the thug of the Internet'. MuckRock (2013). https://www.muckrock.com/news/archives/2013/jan/23/businesses-yelp-thug-of-the-internet.
- Inkoo Kang. 2013. Read nearly 700 FTC complaints regarding Yelp. MuckRock (Jan 2013). https://www.muckrock.com/news/archives/2013/jan/23/businesses-yelp-thug-of-the-internet/
- Ronald B Mincy. 1993. The Urban Institute audit studies: Their research and policy context. In Clear and convincing evidence: Measurement of discrimination in America, M. Fix and R. J. Struyk (Eds.). Urban Institute Press, 165--86.
- Andy Ozment. 2004. Bug auctions: Vulnerability markets reconsidered. In Third Workshop on the Economics of Information Security. 19--26.
- Charles C. Ragin and Howard S. Becker. 1992. What is a case?: Exploring the foundations of social inquiry. Cambridge University Press.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014).
- Satviki Sanjay. 2020. Twitter's AI Revealed to Have Racial Bias After a Viral Photo Experiment. Vice (Sep 2020). https://www.vice.com/en/article/93547v/twitter-ai-algorithm-has-racial-bias
- James C Scott. 1985. Weapons of the weak: Everyday forms of peasant resistance. Yale University Press.
- Nick Seaver. 2019. Knowing algorithms. Digital STS (2019), 412--422.
- Ellen Simpson and Bryan Semaan. 2021. For you, or For ?you"? Everyday LGBTQ+ encounters with TikTok. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1--34.
- Lucy A Suchman. 1987. Plans and situated actions: The problem of human-machine communication. Cambridge University Press.
- Harry Tucker. 2016. Australian Uber drivers say the company is manipulating their ratings to boost its fees. Businessinsider (May 2016). https://www.businessinsider.com.au/australian-uber-drivers-say-the-company-ismanipulating-their-ratings-to-boost-the-companys-fees-2016--5
- Colin Vandenhof and Edith Law. 2019. Contradict the Machine:AHybrid Approach to Identifying Unknown Unknowns. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. 2238--2240.
- Julia Velkova and Anne Kaun. 2019. Algorithmic resistance: Media practices and the politics of repair. Information, Communication & Society (2019), 1--18.
- Karl E Weick. 1995. Sensemaking in organizations. Vol. 3. Sage.
Cited by
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Scholars have developed harm taxonomies related to system misbehaviors and failures [18, 198], particularly to aid algorithmic auditing [177].”
BackgroundLaine, J., Minkkinen, M., & Mäntymäki, M. (2024) · Information & Management · doi:10.1016/j.im.2024.103969 · Cited by 219
“…ethical principles to highlight the benefits that the other principles can elicit, e.g., privacy and accountability [123], transparency [82], or other technical and legal benefits for sys-tem designers, operators, auditors, regulators, and end-users that come from improved accountability [132,135].”
InfluentialAuditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“Further, by providing a basis on which technology providers can be held accountable, audits incentivise investments in adequate risk management (Shen et al., 2021).”
BackgroundKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143
“(2021) have called “everyday algorithm auditing” : the ways that users detect, understand, and interrogate machine behaviors via their day-to-day interactions with algorithmic systems [55].”
BackgroundSelf-citationSupporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604712 · Cited by 143
“Recent works (DeVos et al., 2022; Cabrera et al., 2022; Shen et al., 2021) have operationalized this model to analyse human-driven auditing.”
BackgroundAuditing Algorithms: Understanding Algorithmic Systems from the Outside In
Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000083 · Cited by 140
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S. K., Kapania, S., Terry, M. A., & Wilcox, L. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 137
“For instance, we see UX practitioners using user testing sessions to conduct adversarial testing of potential model harms; prior literature has suggested crowdsourcing [77] or using “bias bounties” 12 [49] or “crowd audits” to identify potentially harmful model outputs [35, 91].”
BackgroundMethodSeeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
Wong, R. Y., Madaio, M. A., & Merrill, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579621 · Cited by 135
Lam, M. S., Gordon, M., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 127
“End users, on the other hand, do possess this rich situated knowledge of the particular impacts that algorithmic systems have on their own communities, and they already identify problematic behaviors of algorithmic systems through their everyday interactions with such systems [66].”
BackgroundResultInfluentialConfronting Power and Corporate Capture at the FAccT Conference
Young, M., Katell, M. A., & Krafft, P. M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533194 · Cited by 111
“We argue that scholarship and other contributions reflecting direct experience of violence and harm should be central to a conference dedicated to understanding and addressing algorithmic harms; particularly as some of these harms become evident only at scale [58].”
BackgroundDeng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
“In addition, expert-led audits may fail to detect certain harmful algorithmic behaviors because these behaviors only arise—or are only recognized as harm-ful—when a system is used in particular context or in particular ways, which auditors may fail to anticipate [22, 30, 34, 42, 79, 80].”
BackgroundMethodInfluentialSelf-citationDeng, W. H., Yildirim, N., Chang, M., Eslami, M., Holstein, K., & Madaio, M. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594037 · Cited by 103
“users’ day to day experience when interacting with the AI systems [26, 28, 79].”
BackgroundSelf-citationInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
“Human-in-the-loop auditing processes leverage everyday users to provide further protection once a system is deployed [63], and we heard from participants in both of our studies that they believed IMCs would be valuable for record-keeping and organizational alignment throughout deployment.”
BackgroundExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
“Toolkit developers interested in pursuing this vision may be able to draw lessons from other open-source community-building efforts [21, 47], or from recent work exploring ways to support collective algorithm auditing [6, 22, 24, 97].”
BackgroundSelf-citationDeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
“However, methods in this taxonomy often fail to surface serious issues that everyday users of algorithmic systems are quickly able to detect once a system is deployed in the wild [71].”
BackgroundMethodInfluentialSelf-citationPower and Public Participation in AI
Corbett, E., Denton, E., & Erete, S. L. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623228 · Cited by 97
“The performance and [15], [16], [39], [59], [60], [63], [66], [67] Rung 3: Informing [9], [11], [38], [40], [ behavior of an AI system can change over time, and even impact the state of the world in ways that were not measurable at the time of deployment [24, 28].”
Background”Because AI is 100% right and safe”: User Attitudes and Sources of AI Authority in India
Kapania, S., Siy, O., Clapper, G., SP, A. M., & Sambasivan, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517533 · Cited by 90
“[115] propose the concept of everyday auditing through which users detect and interrogate problematic biases/behaviors through their everyday interactions with algorithmic systems.”
BackgroundDelmonaco, D., Mayworm, S., Thach, H., Guberman, J., Augusta, A., & Haimson, O. L. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637431 · Cited by 90
“Many of these attempts to detect shadowbanning fall under “everyday algorithmic auditing” in which users attempt to detect and understand potentially problematic interactions with algorithmic systems [119].”
BackgroundStereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models
Ungless, E. L., Roß, B., & Lauscher, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.502 · Cited by 84
Cruising Queer HCI on the DL: A Literature Review of LGBTQ+ People in HCI
Taylor, J., Simpson, E., Tran, A.-T., Brubaker, J. R., Fox, S., & Zhu, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642494 · Cited by 83
They shall be fair, transparent, and robust: auditing learning analytics systems
Simbeck, K. (2023) · AI and Ethics · doi:10.1007/s43681-023-00292-7 · Cited by 82
“This includes methodologies described in academic audits [21, 92], commissioned or voluntary audits [76, 109], user audits [53, 94], and non-binding proposals for standards from institutions [16, 26, 44, 46].”
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Rakova, B., & Dobbe, R. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594014 · Cited by 80
“[106] conduct a field scan of examples where users have meaningfully contributed to third-party audits across particular AI systems including Google Image Search, Google Maps, YouTube, TikTok, and others.”
BackgroundSupporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
“Recent works (DeVos et al., 2022; Cabrera et al., 2022; Shen et al., 2021) have operationalized this model to analyse human-driven auditing.”
BackgroundRismani, S., Shelby, R. M., Smart, A., Jatho, E. W., Kroll, J. A., Moon, A., & Rostamzadeh, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 78
Ehsan, U., Singh, R., Metcalf, J., & Riedl, M. O. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533186 · Cited by 73
A Systematic Literature Review of Human-Centered, Ethical, and Responsible AI
Tahaei, M., Constantinides, M., Quercia, D., & Muller, M. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05284 · Cited by 72
Berman, G., Goyal, N., & Madaio, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 72
“As part of this efort, a range of tools have been developed for use by AI practitioners [e.g., 126, 132, 166], stakeholders [e.g., 59, 122, 166, 170, 177, 180], and community members [e.g., 44, 46, 119, 183], to support RAI at all stages of the AI development and deployment pipeline.”
MethodInfluentialSolyst, J., Yang, E., Xie, S., Ogan, A., Hammer, J., & Eslami, M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610213 · Cited by 68
“[63] suggests that adult users can effectively detect subpar behavior and algorithmic bias in their frequent interactions with AI.”
BackgroundSelf-citationSociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
Lam, M. S., Pandit, A., Kalicki, C. H., Gupta, R., Sahoo, P., & Metaxa, D. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610209 · Cited by 68
“Recently, more audits have started to include end user perspectives, for example by collecting data directly from users (a method known as crowdsourced or collaborative auditing) [35, 51, 63], or by empowering users themselves to run their own audits [33, 66].”
BackgroundFan, X., Xiao, Q., Zhou, X., Pei, J., Sap, M., Lu, Z., & Shen, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713477 · Cited by 65
Self-citationThe Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design
Shen, H., Wang, L., Deng, W. H., Brusse, C., Velgersdijk, R., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533110 · Cited by 64
“They have also proposed methods and frameworks for developers, researchers and everyday users to detect, understand and interrogate problematic machine behaviors [2, 15, 57].”
Self-citationChoi, Y., Kang, E. J., Lee, M. K., & Kim, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581386 · Cited by 64
“suggested a user-centered algorithmic auditing concept where users can scrutinize how algorithms work through day-to-day interactions [63].”
BackgroundWhat’s “Up Next”? Investigating Algorithmic Recommendations on YouTube Across Issues and Over Time
Matamoros-Fernández, A., Gray, J. E., Bartolo, L., Burgess, J., & Suzor, N. P. (2021) · Media and Communication · doi:10.17645/mac.v9i4.4184 · Cited by 63
Chuan, C., Sun, R., Tian, S., & Tsai, W. S. (2024) · Telematics and Informatics · doi:10.1016/j.tele.2024.102135 · Cited by 63
Auditing YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles
Srba, I., Moro, R., Tomlein, M., Pecher, B., Simko, J., Stefancova, E., Kompan, M., Hrckova, A., Podrouzek, J., Gavornik, A., et al. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3568392 · Cited by 61
“[43] introduced an idea of everyday algorithm auditing, in which users detect, understand, and interrogate problematic machine behaviors via their day-to-day interactions with algorithmic systems.”
BackgroundKingsley, S., Sinha, P., Wang, C., Eslami, M., & Hong, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555149 · Cited by 61
Self-citationTowards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 61
“However, harmful algorithmic behavior is not limited to biases encoded in the ML life cycle [159].”
BackgroundMethodInfluentialKarusala, N., Upadhyay, S., Veeraraghavan, R., & Gajos, K. Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641898 · Cited by 57
“A body of work also looks at how to support contestation of systems themselves through auditing [25, 27, 90, 103] and critical scholar-ship and community advocacy [32, 51, 58].”
BackgroundMethodIssar, S. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2233138 · Cited by 56
“…et al., 2022), this study ’ s results suggest that, in the case of TikTok, trust in the institution that owns and operates the platform (ie, ByteDance) represents a more far-reaching sociological issue that warrants inquiry beyond the scope of “ everyday algorithm auditing ” (Shen et al., 2021).”
Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications
Morales‐Navarro, L., Kafai, Y. B., Konda, V., & Metaxa, D. (2024) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3628516.3655752 · Cited by 55
“Studies have looked at how users engage in emergent, everyday auditing practices, without the participation of experts, on social media platforms [52].”
BackgroundWikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia
Kuo, T.-S., Halfaker, A., Cheng, Z., Kim, J., Wu, M.-H., Wu, T., Holstein, K., & Zhu, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642278 · Cited by 54
Self-citationLi, R., Kingsley, S., Fan, C., Sinha, P., Wai, N., Lee, J., Shen, H., Eslami, M., & Hong, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3582074 · Cited by 49
“Past work [65] suggests that there are dierent types of user-driven audits, spanning across dierent levels of algorithmic expertise, collectiveness, and organicness.”
BackgroundMethodResultInfluentialSelf-citationDisambiguating Algorithmic Bias: From Neutrality to Justice
Edenberg, E., & Wood, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604695 · Cited by 46
“highlight, regular auditing by everyday users who engage with algorithmic systems in real-world social and cultural contexts is crucial because these users are well-situated to detect algorithmic harms that may go unnoticed by design teams [130].”
BackgroundJuneja, P., Bhuiyan, M. M., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580846 · Cited by 44
“, conducting a systematic investigation to determine whether the algorithmic output is aligned with “laws and regulations, societal values, ethical desiderata, or industry standards” [61].”
BackgroundCorbett, E., & Denton, E. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594104 · Cited by 41
“Beyond what we found in FAccT, many other mechanisms for transparency have been explored: documentation [72], art [4], public participation [45], onboarding [17], auditing [82], data intermediation [54], and AI literacy [51].”
Misfitting With AI: How Blind People Verify and Contest AI Errors
Alharbi, R., Lor, P., Herskovitz, J., Schoenebeck, S., & Brewer, R. N. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675659 · Cited by 40
“This incident, along with many other cases of users detecting AI limitations and harms [13, 56, 102], motivated a wealth of prior research to study how users make sense of AI [38, 72, 111].”
Imana, B., Korolova, A., & Heidemann, J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579610 · Cited by 40
“User-driven, collaborative approaches rely on day-to-day observations by users to infer algorithmic behavior of the platforms [13, 24, 31, 88].”
InfluentialToward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI AgentsPreprint
Chen, C., Zhang, Z., Khalilov, I., Guo, B., Gebreegziabher, S. A., Ye, Y., Xiao, Z., Yao, Y., Li, T., & Li, T. J.-J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.17934 · Cited by 40
“Current human-centered evaluations for LLM agents primarily fall under human oversight [14, 17, 24] and user-engaged algorithm auditing [23, 38].”
Law and the Emerging Political Economy of Algorithmic Audits
Terzis, P., Veale, M., & Gaumann, N. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658970 · Cited by 39
“…vetted researchers and trusted flaggers assuming a delegated and task-specific auditing role, auditing work of the kind that civil society and the public at large need in order to understand and scrutinise algorithmic systems and services, is left at the discretion and powers of external auditors.”
BackgroundInvestigating Youth AI Auditing
Solyst, J., Peng, C., Deng, W. H., Pratapa, P., Ogan, A., Hammer, J., Hong, J., & Eslami, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732142 · Cited by 38
“However, much FAccT, HCI, and AI research has suggested the potential psychological harms might exhibited in conducting AI auditing and red-teaming activities [21, 38, 65].”
Self-citationBound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms
Dennler, N. S., Ovalle, A., Singh, A., Soldaini, L., Subramonian, A., Tu, H., Agnew, W. S., Ghosh, A., Yee, K., Peradejordi, I. F., et al. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604682 · Cited by 36
Prabhudesai, S., Kasi, A. P., Mansingh, A., Antar, A. D., Shen, H., & Banović, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713714 · Cited by 35
Self-citationLewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30
“in auditing algorithmic systems [122, 123, 125, 135, 150], this field generally lacks defined standards and records could be incomplete”
BackgroundLearning about Data, Algorithms, and Algorithmic Justice on TikTok in Personally Meaningful Ways
Morales‐Navarro, L., Kafai, Y. B., Nguyen, H., DesPortes, K., Vacca, R., Matuk, C. F., Silander, M., Amato, A., Woods, P. J., Castro, F. B. D., et al. (2024) · Proceedings. · doi:10.22318/icls2024.704174 · Cited by 30
“Recently several studies have investigated how everyday non-expert users can find harmful algorithmic behaviors through everyday auditing (Shen et al., 2021; DeVos, 2022).”
BackgroundLi, W., Kuo, J.-C., Sheng, M., Zhang, P., & Wu, Q. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713241 · Cited by 30
“User-driven algorithm auditing is also leveraged to uncover harmful algorithmic behaviors [16, 75].”
Un-Straightening Generative AI: How Queer Artists Surface and Challenge Model Normativity
Taylor, J., Mire, J., Spektor, F., DeVrio, A., Sap, M., Zhu, H., & Fox, S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732061 · Cited by 29
“At the same time, queer people have resisted harmful algorithmic systems by, for instance, conducting collective audits to bring attention to algorithmic discrimination [101].”
Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Robertson, S., Wang, Z. J., Moritz, D., Kery, M. B., & Hohman, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580790 · Cited by 29
“Developing automatic, expert-driven, and crowd-sourcing methods for identifying failures is an active area of research in machine learning and human-computer interaction [19, 25, 31, 77, 114, 132, 136].”
BackgroundThe Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment
Johnson, N., Moharana, S., Harrington, C., Andalibi, N., Heidari, H., & Eslami, M. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658910 · Cited by 27
“…assessment algorithm under-estimated risks posed by perpetrators, withholding necessary protection services that led to adverse outcomes in domestic violence cases. its impacts, (3) dissemination , where these actors amplified their concerns to a wider audience, (4) dialogue between critics of…”
BackgroundSelf-citationWeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
Deng, W. H., Claire, W., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757702 · Cited by 27
“…end users in iteratively investigating, deliberating on, and reporting perceived harms and biases in generative AI (GenAI) systems through support for comparing and making sense of different AI generated outputs (a, b); support for reflection around their own and others’ auditing findings…”
Self-citationBerghout, E. W., Fijneman, R., Hendriks, L., de Boer, M., & Butijn, B.-J. (2022) · Progress in IS · doi:10.1007/978-3-031-11089-4 · Cited by 27
Intersectional Experiences of Unfair Treatment Caused by Automated Computational Systems
van Nuenen, T., Such, J. M., & Coté, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555546 · Cited by 26
“It has been pointed out that everyday users of algorithmic systems are often best poised to detect and raise awareness about harmful behaviors that they encounter in the course of their everyday interactions with these systems [104].”
BackgroundLassiter, T. B., & Fleischmann, K. R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686963 · Cited by 26
“Most of the previous literature focuses on different kinds of audits, the benefits of each, the quality of audits, and how they should be conducted [1, 5, 6, 8, 10, 11].”
Shedding Light on ShadowbanningPreprint
Nicholas, G. (2022) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/xcz2t · Cited by 26
Wang, L., Yurechko, K., Dani, P., Chen, Q. Z., & Zhang, A. X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713691 · Cited by 25
“We further describe our system implementations and experiment design below.”
Xie, S., Zimmerman, J., & Eslami, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713841 · Cited by 24
Self-citationXiao, Q., Zheng, Y., Fan, X., Zhang, B., & Lu, Z. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713279 · Cited by 24
“This user-driven approach can enhance algorithmic transparency and understandability from the user’s perspective [18, 40].”
Kim, S., Bai, Y., Zhu, H., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711079 · Cited by 24
“…existing societal inequities and biases, there are efforts to mitigate bias and create equitable solutions by developing new technologies or making modifications to existing ones - for example, auditing algorithms [41, 132] or deploying new technologies that identify risks and biases [154].”
Self-citationNoh, D. J., Fields, D. A., Morales‐Navarro, L., Cabrera-Sutch, A. M., Kafai, Y. B., & Metaxa, D. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728849 · Cited by 24
“Everyday end-users can initiate audits with or without expert involvement, identifying harmful bias in algorithmic systems [45].”
Terms-we-serve-with: Five dimensions for anticipating and repairing algorithmic harm
Rakova, B., Shelby, R. M., & Ma, M. (2023) · Big Data & Society · doi:10.1177/20539517231211553 · Cited by 24
“This may include the proactive inclusion of collective forms of refusal into user agreements, for example, bug bounty programs to address performance failures (Kenway et al., 2022) and community-led audit studies (Matias et al., 2015; Shen et al., 2021).”
BackgroundLeveraging AI on Social Media & Digital Platforms to Enhance English Language Teaching and Learning
Emon, M. M. H. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1952-0.ch007 · Cited by 24
Algorithm Assurance: Auditing Applications of Artificial Intelligence
Boer, A., de Beer, L., & van Praat, F. (2022) · Progress in IS · doi:10.1007/978-3-031-11089-4_7 · Cited by 24
(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
Jin, A., & Salehi, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641902 · Cited by 23
“…demonstrate the valuable insights that public defenders develop through their everyday encounters with CFS in the U.S. criminal legal system, further motivating growing eforts in HCI to engage downstream stakeholders in designing performance evaluations of AI systems (e.g., [28, 55, 76, 81]).”
Supporting User Engagement in Testing, Auditing, and Contesting AI
Deng, W. H., Lam, M. S., Cabrera, Á. A., Metaxa, D., Eslami, M., & Holstein, K. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611279 · Cited by 23
Self-citationAlgorithmic Regimes: Methods, Interactions, and Politics
Jarke, J., Prietl, B., Egbert, S., Boeva, Y., Heuer, H., & Arnold, M. (2024) · Amsterdam University Press eBooks · doi:10.1515/9789048556908 · Cited by 23
Okoronkwo, C. E. (2024) · International Journal of Social Sciences and Management Review · doi:10.37602/ijssmr.2024.7523 · Cited by 22
“While some scholars (e.g., Daneshjou et al., 2021; Norori et al., 2021) have asked for more inclusive data sets to be utilized in algorithm training, others (e.g., Bandy, 2021; Shen et al., 2021) have argued for algorithm audits to evaluate the impact of these systems on excluded groups.”
BackgroundA Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Harvey, E., Kizilcec, R. F., & Koenecke, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732137 · Cited by 22
“While we emphasize that a gold standard approach would involve a collective audit [82], we nevertheless recognize that this may be prohibitively expensive or time-consuming, especially for external auditors, and therefore propose dialect transformation tools like Multi-VALUE as an alternative.”
Black, E., Gillis, T. B., & Hall, Z. Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658928 · Cited by 22
“…raised in recent algorithmic auditing literature, namely, concerns and limitations around internal audits [24, 66], calls for assessment and monitoring throughout model deployment [17], and the possibility for model developers and systems users to work together to assess an algorithmic system [73].”
BackgroundBuilding, Shifting, & Employing Power: A Taxonomy of Responses From Below to Algorithmic Harm
DeVrio, A., Eslami, M., & Holstein, K. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658958 · Cited by 21
“…real-world case studies indicate that responses “from below”—i.e., from members of the public or others without direct power to make decisions about the design or deployment of algorithmic systems [142]—can be effective in garnering power and working toward harm remediation [45, 85, 120, 137, 142].”
BackgroundInfluentialSelf-citationFace Work: A Human-Centered Investigation into Facial Verification in Gig Work
Watkins, E. A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579485 · Cited by 21
“…find consistent evidence that these workers do their own research to reverse engineer the algorithms of their work, modifying their own behavior to create a platform environment better optimized for the worker [9,21,43,62], and identify errors in systems [72], in a form of “ folk tradecraft ” [91].”
BackgroundLópez, C., Davidoff, A., Luco, F., Humeres, M., & Correa, T. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2356910 · Cited by 21
“For example, users with more negative attitudes toward algorithms, especially those from marginalized groups who are adversely affected by their biases, may develop strategies as forms of resistance against these negative impacts (Cobbe, 2021; Shen et al., 2021).”
Gausen, A., Mitra, B., & Lindley, S. E. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658900 · Cited by 21
Investigating What Factors Influence Users’ Rating of Harmful Algorithmic Bias and Discrimination
Kingsley, S., Zhi, J., Deng, W. H., Lee, J. E., Zhang, S. Q., Eslami, M., Holstein, K., Hong, J., Li, T., & Shen, H. (2024) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v12i1.31602 · Cited by 21
“For instance, experts might lack the relevant cultural knowledge and lived experience to recognize and know where to look for certain kinds of harmful behaviors (DeVos et al. 2022; Holstein et al. 2019; Shen et al. 2021; Deng et al. 2022).”
BackgroundMethodInfluentialSelf-citationExperiences of Censorship on TikTok Across Marginalised Identities
Ungless, E. L., Markl, N., & Roß, B. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35912 · Cited by 20
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AIPreprint
Deng, W. H., Wang, C., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.01397 · Cited by 20
“…end users in iteratively investigating, deliberating on, and reporting perceived harms and biases in generative AI (GenAI) systems through support for comparing and making sense of different AI generated outputs (a, b); support for reflection around their own and others’ auditing findings…”
Self-citationPolicy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
Lam, M. S., Hohman, F., Moritz, D., Bigham, J. P., Holstein, K., & Kery, M. B. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747680 · Cited by 18
“In response, researchers have introduced human-centered evaluation approaches such as proactively envisioning user-facing harms [11, 52, 64, 69, 92] and engaging diverse stakeholders in evaluations [50, 99], audits [18, 21, 51, 80], and red-teaming efforts [94].”
Self-citationKollig, F., Pater, J. A., Nova, F. F., & Fiesler, C. L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713322 · Cited by 18
“17 Bottom-up scrutiny of AI-moderated systems, in the form of everyday algorithmic auditing, has been argued to be highly efective at catching harmful outcomes missed by other measures [136].”
Impact Assessment Card: Communicating Risks and Benefits of AI Uses
Bogucka, E. P., Constantinides, M., Šćepanović, S., & Quercia, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757482 · Cited by 18
“These barriers highlight the need for resources that not only simplify complex AI concepts but also actively engage non-technical actors in the broader governance ecosystem [21, 80].”
Clarify: Improving Model Robustness With Natural Language Corrections
Lee, Y., Lam, M. S., Vasconcelos, H., Bernstein, M. S., & Finn, C. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676362 · Cited by 18
The Cultivated Practices of Text-to-Image Generation
Oppenlaender, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-66528-8_14 · Cited by 18
Building Solidarity Amid Hostility: Experiences of Fat People in Online Communities
Payne, B. H., Taylor, J., Spiel, K., & Fiesler, C. L. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701186 · Cited by 18
“For instance, research drawing on sensemaking [105] — the ongoing process of trying to construct a working understanding of the world — has explored how people collectively understand their health [83, 137] and audit algorithmic systems [115].”
Morales‐Navarro, L., Kafai, Y. B., Vogelstein, L., Yu, E., & Metaxa, D. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i28.35192 · Cited by 18
Tang, X., Lima, G., Jiang, L., Simko, L., & Zou, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710935 · Cited by 17
“• At the temporal layer, a critical concern with AI is the unpredictability of the harms it may cause [9, 92], which often emerge through users’ everyday interactions with these systems [177].”
How public involvement can improve the science of AI
Matias, J. N., & Price, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2421111122 · Cited by 17
Ramesh, D., Henning, C., Escher, N., Zhu, H., Lee, M. K., & Banović, N. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596004 · Cited by 17
“Platform “afordances” such as API endpoints that can help researchers systematically observe the behavior of algorithms may be missing [65].”
BackgroundEslami, M., Fox, S., Shen, H., Fan, B., Lin, Y., Farzan, R., & Schwanke, B. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732173 · Cited by 17
Self-citationInternational AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
Designing for Difference: How We Learn to Stop Worrying and Love the Doppelganger
Seberger, J. S., & Gupta, S. D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713560 · Cited by 16
“We do so to offer a productive foil to the discourses of surveillance [40], control [50], and creepiness [63, 69] that birthed the terms, and which create pervasive conditions of cruel optimism [12, 13] at the scale of the “data subject” (e.g., [70, 88]).”
PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-TeamingPreprint
Deng, W. H., Kim, S. S. Y., Jha, A., Holstein, K., Eslami, M., Wilcox, L. G., & Gatys, L. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.03728 · Cited by 16
“In addition, we found that successful prompts in RP + PG Users contain attack style rooted in storytelling and persuasion, which may reflect how everyday AI users often frame prompts in more narrative-driven or conversational ways Shen et al. [2021], DeVos et al. [2022], Lam et al. [2022].”
Self-citationAccess Denied: Meaningful Data Access for Quantitative Algorithm Audits
Zaccour, J., Binns, R., & Rocher, L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713963 · Cited by 14
“For user-facing algorithms, user-driven audit frameworks have been proposed [43, 87, 118] and are increasingly adopted by practitioners [41].”
Influential
Metadata
- Authors
- Shen, H., DeVos, A., Eslami, M., & Holstein, K.
- Affiliations
- Carnegie Mellon University (US)
- Carnegie Mellon University, Pittsburgh, PA, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3160887439
- Cited by
- 171
- References
- 49
- Citations by year
- 2026: 43 · 2025: 49 · 2024: 28 · 2023: 31 · 2022: 19 · 2021: 1
- FWCI
- 24.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (IIS-2040942)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- algorithmic auditing, algorithmic bias, platform design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.