Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
The DOI matches its Crossref record; 130 of the 130 references it cites were found in a registry or an index.
Abstract
While algorithm audits are growing rapidly in commonality and public importance, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to do just that, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). The review shows how previous audit studies have exposed public-facing algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. Based on the studies reviewed, it also suggests some behaviors (e.g. discrimination on the basis of intersectional identities), domains (e.g. advertising algorithms), methods (e.g. code auditing), and organizations (e.g. Twitter, TikTok, LinkedIn) that call for future audit attention. The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research working toward algorithmic justice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The challenges associated with selecting subgroups for fairness assessment are well-documented by Raji et al. [52] and Barocas et al. [5], who note that intersectional and/or rare groups are often hard to identify and underrepresented in the data.”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [5]
In the citing paper
“and Bandy (2021) pointed out that, in the absence of agreed standards, technology providers' claims that their AI systems have been audited are hard to verify.”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Bandy, 2021
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
(2018) · Cited by 3,510
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Big other: Surveillance Capitalism and the Prospects of an Information Civilization
Zuboff, S. (2015) · Journal of Information Technology · doi:10.1057/jit.2015.5 · Cited by 3,205
The New Jim Crow: Mass Incarceration in the Age of Colorblindness
Hauhart, R. C. (2011) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306111419111 · Cited by 2,732
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Research works printed but not linked: 32
- Penelope Muse Abernathy. 2018. The Expanding News Desert. University of North Carolina Press Chapel Hill. https://www.cislm.org/wp-content/uploads/2018/10/The-Expanding-News-Desert-10_14-Web.pdf
- Michael Armbrust Ion Stoica Matei Zaharia Armando Fox Rean Griffith Anthony D. Joseph Randy Katz Andy Konwinski Gunho Lee David Patterson and Ariel Rabkin. 2018. It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process. https://acm-fca.org/2018/03/29/negativeimpacts/
- Jack Bandy and Nicholas Diakopoulos. 2020. Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News. International conference on web and social media (ICWSM) (2020).
- Pinar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. 2019. Social B(eye)as: Human and machine descriptions of people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 583--591. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3255/3123
- Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and Society.
- Ruha Benjamin. 2019. Race After Technology. Polity Press.
- José González Cabañas, Ángel Cuevas, and Rubén Cuevas. 2018. Unveiling and Quantifying Facebook Exploitation of Sensitive Personal Data for Advertising Purposes. In USENIX Security Symposium. 479--495. https://www.usenix.org/conference/usenixsecurity18/presentation/cabanas
- 10.4324/9781315582924--10 · doi:10.4324/9781315582924--10
- 10.1093/joc · doi:10.1093/joc
- Amit Datta, Anupam Datta, Jael Makagon, Deirdre K Mulligan, and Michael Carl Tschantz. 2018. Discrimination in Online Advertising: A Multidisciplinary Inquiry. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81 (2018), 20--34. http://proceedings.mlr.press/v81/datta18a/datta18a.pdf
- Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52---59. arxiv: 1906.02659 · arXiv
- Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. Digital Dominance: The Power of Google, Amazon, Facebook, and Apple (2018).
- Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
- Editors. 2020. Price Gouging Laws by State. https://consumer.findlaw.com/consumer-transactions/price-gouging-laws-by-state.html
- Elsevier. 2016. Scopus | The largest database of peer-reviewed literature.
- Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. "Be careful; Things can be worse than they appear" - Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 62--71. https://ojs.aaai.org/index.php/ICWSM/article/view/14898
- Anthony Flores, Christopher T. Lowenkamp, and Kristin Bechtel. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias". HeinOnline (2016).
- 10.7551/mitpress · doi:10.7551/mitpress
- 10.1093/bjsw · doi:10.1093/bjsw
- Shan Jiang, Ronald E. Robertson, and Christo Wilson. 2019 b. Bias misperceived: The role of partisanship and misinformation in YouTube comment moderation. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, Vol. 13(01). 278--289. https://ojs.aaai.org/index.php/ICWSM/article/view/3229
- Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. http://proceedings.mlr.press/v80/kearns18a.html
- Kyriakos Kyriakou, Pinar Barlas, Styliani Kleanthous, and Jahna Otterbacher. 2019. Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 313--322. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3232/3100
- Emma Lurie and Eni Mustafaraj. 2019. Opening Up the Black Box: Auditing Google's Top Stories Algorithm. In Proceedings of the International Florida Artificial Intelligence Research Society Conference. https://par.nsf.gov/biblio/10101277
- Connor Mcmahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. In Proceedings of the 11th International AAAI Conference on Web and Social Media. https://ojs.aaai.org/index.php/ICWSM/article/view/14883
- 10.1145/2535372.2535415 · DOI · Paper page
- Safiya Noble. 2013. Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible Culture (2013).
- 10.1145/3292522.3326047 · DOI
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and discrimination: converting critical concerns into productive inquiry.
- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2020. The risk of racial bias in hate speech detection. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 1668--1678.
- 10.2139/ssrn.2208240 · DOI
- Nicholas Vincent, Isaac Johnson, Patrick Sheehan, and Brent Hecht. 2019 b. Measuring the Importance of User-Generated Content to Search Engines. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). https://ojs.aaai.org/index.php/ICWSM/article/view/3248
- Sara Wachter-Boettcher. 2017. Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sexism. WW Norton & Company.
Cited by
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Bandy [18] summarizes four problematic behaviors of algorithmic systems — discrimination, distortion, exploitation, and misjudgement.”
BackgroundMilli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“Sloane (2021) argued that audits commissioned by technology providers are insufficiently independent, and Bandy (2021) pointed out that, in the absence of agreed standards, technology providers’ claims that their AI systems have been audited are hard to verify.”
BackgroundA survey on bias in visual datasets
Fabbrizzi, S., Papadopoulos, S., Ntoutsi, E., & Kompatsiaris, I. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103552 · Cited by 137
Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
Stray, J., Halevy, A., Assar, P., Hadfield-Menell, D., Boutilier, C., Ashar, A., Bakalar, C., Beattie, L., Ekstrand, M., Leibowicz, C., et al. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3632297 · Cited by 122
An Empirical Investigation of Personalization Factors on TikTok
Boeker, M., & Urman, A. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512102 · Cited by 284
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Tulio Ribeiro, M., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604712 · Cited by 78
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
Transparency and accountability in digital public services: Learning from the Brazilian cases
Saldanha, D. M. F., Dias, C. N., & Guillaumon, S. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101680 · Cited by 156
Lam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 77
Machine Learning Techniques for Sentiment Analysis of COVID-19-Related Twitter Data
Braig, N., Benz, A., Voth, S., Breitenbach, J., & Buettner, R. (2023) · IEEE Access · doi:10.1109/access.2023.3242234 · Cited by 112
Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
Vecchione, B., Levy, K., & Barocas, S. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483294 · Cited by 51
Mökander, J., & Axente, M. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01286-x · Cited by 62
Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions
Ho, J. Q. H., Hartanto, A., Koh, A., & Majeed, N. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100145 · Cited by 111
Auditing YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles
Srba, I., Moro, R., Tomlein, M., Pecher, B., Simko, J., Stefancova, E., Kompan, M., Hrckova, A., Podrouzek, J., Gavornik, A., et al. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3568392 · Cited by 61
“To do so, we employ a sock puppet auditing methodology [5, 42].”
BackgroundMethodLamers, L., Meijerink, J., & Rettagliata, G. (2024) · Human Resource Management · doi:10.1002/hrm.22204 · Cited by 78
Hartmann, D., de Pereira, J. R. L., Streitbörger, C., & Berendt, B. (2025) · AI and Ethics · doi:10.1007/s43681-024-00595-3 · Cited by 58
News recommender systems: a programmatic research review
Mitova, E., Blassnig, S., Strikovic, E., Urman, A., Hannák, A., de Vreese, C. H., & Esser, F. (2022) · Annals of the International Communication Association · doi:10.1080/23808985.2022.2142149 · Cited by 82
Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 75
How Do Users Experience Moderation?: A Systematic Literature Review
Ma, R., You, Y., Gui, X., & Kou, Y. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610069 · Cited by 62
Challenges of Moderating Social Virtual Reality
Sabri, N., Chen, B., Teoh, A., Dow, S. P., Vaccaro, K., & ElSherief, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581329 · Cited by 61
Yampolskiy, R. V. (2024) · AI and Ethics · doi:10.1007/s43681-024-00420-x · Cited by 80
Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
Lam, M. S., Pandit, A., Kalicki, C. H., Gupta, R., Sahoo, P., & Metaxa, D. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610209 · Cited by 39
Law and the Emerging Political Economy of Algorithmic Audits
Terzis, P., Veale, M., & Gaumann, N. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658970 · Cited by 39
Sloane, M., & Zakrzewski, J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533156 · Cited by 61
The importance of algorithm skills for informed Internet use
Gruber, J., & Hargittai, E. (2023) · Big Data & Society · doi:10.1177/20539517231168100 · Cited by 62
Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications
Morales‐Navarro, L., Kafai, Y. B., Konda, V., & Metaxa, D. (2024) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3628516.3655752 · Cited by 55
An external stability audit framework to test the validity of personality prediction in AI hiring
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., Khan, F. A., & Stoyanovich, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00861-0 · Cited by 57
Juneja, P., Bhuiyan, M. M., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580846 · Cited by 44
A Sociotechnical Audit: Assessing Police Use of Facial Recognition
Radiya-Dixit, E., & Neff, G. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594084 · Cited by 45
Towards a Better Understanding of Evaluating Trustworthiness in AI Systems
Kemmerzell, N., Schreiner, A., Khalid, H., Schalk, M., & Bordoli, L. (2025) · ACM Computing Surveys · doi:10.1145/3721976 · Cited by 45
Rejeb, A., Rejeb, K., Appolloni, A., & Treiblmaier, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16768-x · Cited by 40
Kyriakou, K., & Otterbacher, J. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00092-2 · Cited by 38
Intersectional Experiences of Unfair Treatment Caused by Automated Computational Systems
van Nuenen, T., Such, J. M., & Coté, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555546 · Cited by 26
Imana, B., Korolova, A., & Heidemann, J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579610 · Cited by 40
Artificial intelligence and algorithmic bias? Field tests on social network with teens
Cecere, G., Jean, C., Guel, F. L., & Manant, M. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123204 · Cited by 25
Challenges in Understanding Human-Algorithm Entanglement During Online Information Consumption
Lewandowsky, S., Robertson, R. E., & DiResta, R. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231180809 · Cited by 37
Auditing GPT's Content Moderation Guardrails: Can ChatGPT Write Your Favorite TV Show?
Mahomed, Y., Crawford, C. M., Gautam, S., Friedler, S. A., & Metaxa, D. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658932 · Cited by 38
Makhortykh, M., Urman, A., & Wijermars, M. (2022) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-94 · Cited by 34
Youths' Perceptions of Data Collection in Online Advertising and Social Media
Goray, C., & Schoenebeck, S. Y. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555576 · Cited by 25
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., & Stoyanovich, J. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534189 · Cited by 26
Ye, J., Luceri, L., & Ferrara, E. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732159 · Cited by 74
Accessing the Impact of TikTok’s Algorithm on Regional Inequality in Health Information
Li, J., & Shi, W. (2024) · Health Communication · doi:10.1080/10410236.2024.2414882 · Cited by 22
Noh, D. J., Fields, D. A., Morales‐Navarro, L., Cabrera-Sutch, A. M., Kafai, Y. B., & Metaxa, D. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728849 · Cited by 24
Public sector AI transparency standard: UK Government seeks to lead by example
Kingsman, N., Kazim, E., Chaudhry, M. A., Hilliard, A., Koshiyama, A., Polle, R., Pavey, G., & Mohammed, U. (2022) · Discover Artificial Intelligence · doi:10.1007/s44163-022-00018-4 · Cited by 23
Bubbles bursting: Investigating and measuring the personalisation of social media searches
Yang, C., Xu, X., Nunes, B. P., & Siqueira, S. W. M. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.101999 · Cited by 29
Lassiter, T. B., & Fleischmann, K. R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686963 · Cited by 26
Job Recommender Systems: A ReviewPreprint
de Ruijt, C., & Bhulai, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13576 · Cited by 87
Gaebler, J. D., Goel, S., Huq, A., & Tambe, P. (2024) · Behavioral Science & Policy · doi:10.1177/23794607251320229 · Cited by 15
Hartmann, D., Oueslati, A., Staufer, D., Pohlmann, L., Munzert, S., & Heuer, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713998 · Cited by 53
Prabhudesai, S., Kasi, A. P., Mansingh, A., Antar, A. D., Shen, H., & Banović, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713714 · Cited by 35
Kim, S., Bai, Y., Zhu, H., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711079 · Cited by 24
From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing
Cen, S. H., & Alur, R. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694711 · Cited by 56
Verwiebe, R., Buder, C., Weissmann, S., Osorio-Krauter, C., & Philipp, A. (2024) · New Media & Society · doi:10.1177/14614448241307931 · Cited by 29
Kravets, D., Ryzhova, A., Toepfl, F., & Beseler, A. (2023) · Journalism · doi:10.1177/14648849231157845 · Cited by 19
Shi, W., & Li, J. (2024) · Communication Research · doi:10.1177/00936502241262056 · Cited by 15
Auditing Risk Prediction of Long-Term Unemployment
Seidelin, C., Moreau, T., Shklovski, I., & Møller, N. H. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492827 · Cited by 20
Time to audit your AI algorithms
Sandu, I., Wiersma, M., & Manichand, D. (2022) · Maandblad voor Accountancy en Bedrijfseconomie · doi:10.5117/mab.96.90108 · Cited by 12
Leveraging Knowledge Graphs for AI System Auditing and Transparency
Waltersdorfer, L., & Sabou, M. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100849 · Cited by 27
The right to audit and power asymmetries in algorithm auditing
Urman, A., Smirnov, I., & Lasser, J. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00454-5 · Cited by 12
Gausen, A., Mitra, B., & Lindley, S. E. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658900 · Cited by 21
Wang, S., Huang, S., Zhou, A., & Metaxa, D. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687046 · Cited by 11
Evaluating and Mitigating Discrimination in Language Model Decisions
Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023) · arXiv · doi:10.48550/arxiv.2312.03689 · Cited by 11
“…and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such discrimination.”
BackgroundHarvey, E., Sandhaus, H., Jacobs, A. Z., Moss, E., & Sloane, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642004 · Cited by 18
Shi, W., & Li, J. (2025) · Digital Journalism · doi:10.1080/21670811.2025.2450312 · Cited by 20
Güvenilir Yapay Zeka ve İç Denetim
Ağdeniz, Ş. (2024) · Denetişim · doi:10.58348/denetisim.1384391 · Cited by 15
Reducing Exposure to Harmful Content via Graph Rewiring
Coupette, C., Neumann, S., & Gionis, A. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599489 · Cited by 16
Nigatu, H. H., & Raji, I. D. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658546 · Cited by 36
Perreault, B., Lee, J., Shava, R., & Mustafaraj, E. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658916 · Cited by 14
How Can I Signal You To Trust Me: Investigating AI Trust Signalling in Clinical Self-Assessments
Kollerup, N. K., Wester, J., Skov, M. B., & van Berkel, N. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661612 · Cited by 16
A systematic review of algorithm auditing processes to assess bias and risks in AI systems
Funda, V. (2025) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd11489 · Cited by 17
Imagining the Far East: Exploring Perceived Biases in AI-Generated Images of East Asian Women
Lan, X., An, J., Guo, Y., Chiyou, T., Cai, X., & Zhang, J. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3719678 · Cited by 8
Advocating for a community-centred model for responding to potential information harms
Wardle, C., & Scales, D. A. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02233-2 · Cited by 11
‘I hope this email finds you well’: how synthetic affect circulates through MagicSchool AI
Robinson, B., & Leander, K. M. (2025) · Learning Media and Technology · doi:10.1080/17439884.2025.2527920 · Cited by 8
Learning AI Auditing: A Case Study of Teenagers Auditing a Generative AI Model
Morales‐Navarro, L., Gan, M. A., Yu, E., Vogelstein, L., Kafai, Y. B., & Metaxa, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757620 · Cited by 13
Measuring an artificial intelligence language model’s trust in humans using machine incentives
Johnson, T., & Obradovich, N. (2024) · Journal of Physics Complexity · doi:10.1088/2632-072x/ad1c69 · Cited by 17
Data Portability as a Tool for Audit
Zwiebelmann, Z., & Henderson, T. (2021) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3460418.3479343 · Cited by 9
Steps towards a therapeutic artificial intelligence
McCarty, W. (2024) · Interdisciplinary Science Reviews · doi:10.1177/03080188241233129 · Cited by 8
Law and the Emerging Political Economy of Algorithmic AuditsPreprint
Terzis, P., Veale, M., & Gaumann, N. (2024) · doi:10.31219/osf.io/xvqz7 · Cited by 7
van Hoof, M., Trilling, D., Möller, J. E., & Meppelink, C. S. (2024) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmae020 · Cited by 15
Ranjitsingh, L. M., & Rao, T. V. S. (2025) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-025-02865-7 · Cited by 16
Hagar, N., & Shaw, A. (2022) · Journal of Communication · doi:10.1093/joc/jqac032 · Cited by 6
The Cost of Reach: Testing the Role of Ad Delivery Algorithms in Online Political Campaigns
Votta, F., Dobber, T., Guinaudeau, B., Helberger, N., & de Vreese, C. H. (2024) · Political Communication · doi:10.1080/10584609.2024.2439317 · Cited by 15
Urman, A., Makhortykh, M., & Hannák, A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732026 · Cited by 16
Santini, R. M., Salles, D., & Mattos, B. (2023) · Policy & Internet · doi:10.1002/poi3.380 · Cited by 25
YouTube and Conspiracy Theories: A Longitudinal Audit of Information Panels
Godinez, L., & Mustafaraj, E. (2024) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3648188.3675128 · Cited by 6
Herd Accountability of Privacy-Preserving Algorithms: A Stackelberg Game Approach
Yang, Y.-T. C., Zhang, T., & Zhu, Q. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3540357 · Cited by 11
Decoding the Meaning of Success on Digital Labor Platforms: Worker-Centered Perspectives
Kim, P., Asante-Agyei, C. O., Muñoz, I., Dunn, M., & Sawyer, S. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710957 · Cited by 11
Revisiting Algorithmic Audits of TikTok: Poor Reproducibility and Short-term Validity of Findings
Mosnar, M., Skurla, A., Pecher, B., Tibenský, M., Jakubcik, J., Bindas, A., Sakalik, P., & Srba, I. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730293 · Cited by 18
Why am I seeing this: Democratizing End User Auditing for Online Content Recommendations
Chen, C., Li, L., Cao, L., Ye, Y., Li, T., Yao, Y., & Li, T. J.-J. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747798 · Cited by 17
The Google news showcase dilemma: journalism between regulation and platform power in Indonesia
Sianturi, H. R. P. (2025) · Media Asia · doi:10.1080/01296612.2025.2549555 · Cited by 4
Yang, Y., Zhang, T., & Zhu, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-50670-3_18 · Cited by 8
Saletti, S. M. R., & den Broucke, S. V. (2024) · Current Addiction Reports · doi:10.1007/s40429-024-00576-w · Cited by 5
Analyzing AI Bias: The Discourse of Terror and Sport Ahead of Paris 2024 Olympics
Samuel-Azran, T., Manor, I., Yitzhak, E., & Galily, Y. (2024) · American Behavioral Scientist · doi:10.1177/00027642241261265 · Cited by 9
Who evaluates the algorithms? An overview of the algorithmic accountability ecosystem
Criado, J. I., & Guevara-Gomez, A. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657247 · Cited by 8
Metadata
- Authors
- Bandy, J.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Bandy, Jack
- ORCID
- Bandy, Jack
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-34
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3159435787
- Cited by
- 193
- References
- 127
- Citations by year
- 2026: 31 · 2025: 60 · 2024: 45 · 2023: 32 · 2022: 18 · 2021: 7
- FWCI
- 31.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- algorithmic auditing, algorithmic discrimination, search algorithms, intersectional discrimination, audit procedures, algorithmic justice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.