The DOI matches its Crossref record; 130 of the 130 references it cites were found in a registry or an index.
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Abstract
While algorithm audits are growing rapidly in commonality and public importance, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to do just that, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). The review shows how previous audit studies have exposed public-facing algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. Based on the studies reviewed, it also suggests some behaviors (e.g. discrimination on the basis of intersectional identities), domains (e.g. advertising algorithms), methods (e.g. code auditing), and organizations (e.g. Twitter, TikTok, LinkedIn) that call for future audit attention. The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research working toward algorithmic justice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The challenges associated with selecting subgroups for fairness assessment are well-documented by Raji et al. [52] and Barocas et al. [5], who note that intersectional and/or rare groups are often hard to identify and underrepresented in the data.”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [5]
In the citing paper
“and Bandy (2021) pointed out that, in the absence of agreed standards, technology providers' claims that their AI systems have been audited are hard to verify.”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Bandy, 2021
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
(2018) · Cited by 3,510
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Big other: Surveillance Capitalism and the Prospects of an Information Civilization
Zuboff, S. (2015) · Journal of Information Technology · doi:10.1057/jit.2015.5 · Cited by 3,205
The New Jim Crow: Mass Incarceration in the Age of Colorblindness
Hauhart, R. C. (2011) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306111419111 · Cited by 2,732
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Research works printed but not linked: 33
- Penelope Muse Abernathy. 2018. The Expanding News Desert. University of North Carolina Press Chapel Hill. https://www.cislm.org/wp-content/uploads/2018/10/The-Expanding-News-Desert-10_14-Web.pdf
- Michael Armbrust Ion Stoica Matei Zaharia Armando Fox Rean Griffith Anthony D. Joseph Randy Katz Andy Konwinski Gunho Lee David Patterson and Ariel Rabkin. 2018. It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process. https://acm-fca.org/2018/03/29/negativeimpacts/
- Jack Bandy and Nicholas Diakopoulos. 2020. Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News. International conference on web and social media (ICWSM) (2020).
- Pinar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. 2019. Social B(eye)as: Human and machine descriptions of people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 583--591. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3255/3123
- Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and Society.
- Ruha Benjamin. 2019. Race After Technology. Polity Press.
- José González Cabañas, Ángel Cuevas, and Rubén Cuevas. 2018. Unveiling and Quantifying Facebook Exploitation of Sensitive Personal Data for Advertising Purposes. In USENIX Security Symposium. 479--495. https://www.usenix.org/conference/usenixsecurity18/presentation/cabanas
- 10.4324/9781315582924--10 · doi:10.4324/9781315582924--10
- 10.1093/joc · doi:10.1093/joc
- Amit Datta, Anupam Datta, Jael Makagon, Deirdre K Mulligan, and Michael Carl Tschantz. 2018. Discrimination in Online Advertising: A Multidisciplinary Inquiry. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81 (2018), 20--34. http://proceedings.mlr.press/v81/datta18a/datta18a.pdf
- Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52---59. arxiv: 1906.02659 · arXiv
- Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. Digital Dominance: The Power of Google, Amazon, Facebook, and Apple (2018).
- Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
- Editors. 2020. Price Gouging Laws by State. https://consumer.findlaw.com/consumer-transactions/price-gouging-laws-by-state.html
- Elsevier. 2016. Scopus | The largest database of peer-reviewed literature.
- Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. "Be careful; Things can be worse than they appear" - Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 62--71. https://ojs.aaai.org/index.php/ICWSM/article/view/14898
- Anthony Flores, Christopher T. Lowenkamp, and Kristin Bechtel. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias". HeinOnline (2016).
- 10.7551/mitpress · doi:10.7551/mitpress
- 10.1093/bjsw · doi:10.1093/bjsw
- 10.1080/01972240050133634 · DOI
- Shan Jiang, Ronald E. Robertson, and Christo Wilson. 2019 b. Bias misperceived: The role of partisanship and misinformation in YouTube comment moderation. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, Vol. 13(01). 278--289. https://ojs.aaai.org/index.php/ICWSM/article/view/3229
- Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. http://proceedings.mlr.press/v80/kearns18a.html
- Kyriakos Kyriakou, Pinar Barlas, Styliani Kleanthous, and Jahna Otterbacher. 2019. Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 313--322. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3232/3100
- Emma Lurie and Eni Mustafaraj. 2019. Opening Up the Black Box: Auditing Google's Top Stories Algorithm. In Proceedings of the International Florida Artificial Intelligence Research Society Conference. https://par.nsf.gov/biblio/10101277
- Connor Mcmahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. In Proceedings of the 11th International AAAI Conference on Web and Social Media. https://ojs.aaai.org/index.php/ICWSM/article/view/14883
- 10.1145/2535372.2535415 · DOI · Paper page
- Safiya Noble. 2013. Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible Culture (2013).
- 10.1145/3292522.3326047 · DOI
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and discrimination: converting critical concerns into productive inquiry.
- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2020. The risk of racial bias in hate speech detection. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 1668--1678.
- 10.2139/ssrn.2208240 · DOI
- Nicholas Vincent, Isaac Johnson, Patrick Sheehan, and Brent Hecht. 2019 b. Measuring the Importance of User-Generated Content to Search Engines. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). https://ojs.aaai.org/index.php/ICWSM/article/view/3248
- Sara Wachter-Boettcher. 2017. Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sexism. WW Norton & Company.
Cited by
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
Milli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
Metadata
- Authors
- Bandy, J.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Bandy, Jack
- ORCID
- Bandy, Jack
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-34
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3159435787
- Cited by
- 193
- References
- 127
- Citations by year
- 2026: 31 · 2025: 60 · 2024: 45 · 2023: 32 · 2022: 18 · 2021: 7
- FWCI
- 31.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- algorithmic auditing, algorithmic discrimination, search algorithms, intersectional discrimination, audit procedures, algorithmic justice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. https://app.cochat.ai/papers/10.1145/3449148
- The paper
- Bandy, J. (2021). Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3449148
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.