Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
The DOI matches its Crossref record; 130 of the 130 references it cites were found in a registry or an index.
Abstract
While algorithm audits are growing rapidly in commonality and public importance, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to do just that, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). The review shows how previous audit studies have exposed public-facing algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. Based on the studies reviewed, it also suggests some behaviors (e.g. discrimination on the basis of intersectional identities), domains (e.g. advertising algorithms), methods (e.g. code auditing), and organizations (e.g. Twitter, TikTok, LinkedIn) that call for future audit attention. The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research working toward algorithmic justice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The challenges associated with selecting subgroups for fairness assessment are well-documented by Raji et al. [52] and Barocas et al. [5], who note that intersectional and/or rare groups are often hard to identify and underrepresented in the data.”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [5]
In the citing paper
“and Bandy (2021) pointed out that, in the absence of agreed standards, technology providers' claims that their AI systems have been audited are hard to verify.”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Bandy, 2021
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
(2018) · Cited by 3,510
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Big other: Surveillance Capitalism and the Prospects of an Information Civilization
Zuboff, S. (2015) · Journal of Information Technology · doi:10.1057/jit.2015.5 · Cited by 3,205
The New Jim Crow: Mass Incarceration in the Age of Colorblindness
Hauhart, R. C. (2011) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306111419111 · Cited by 2,732
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Research works printed but not linked: 32
- Penelope Muse Abernathy. 2018. The Expanding News Desert. University of North Carolina Press Chapel Hill. https://www.cislm.org/wp-content/uploads/2018/10/The-Expanding-News-Desert-10_14-Web.pdf
- Michael Armbrust Ion Stoica Matei Zaharia Armando Fox Rean Griffith Anthony D. Joseph Randy Katz Andy Konwinski Gunho Lee David Patterson and Ariel Rabkin. 2018. It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process. https://acm-fca.org/2018/03/29/negativeimpacts/
- Jack Bandy and Nicholas Diakopoulos. 2020. Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News. International conference on web and social media (ICWSM) (2020).
- Pinar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. 2019. Social B(eye)as: Human and machine descriptions of people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 583--591. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3255/3123
- Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and Society.
- Ruha Benjamin. 2019. Race After Technology. Polity Press.
- José González Cabañas, Ángel Cuevas, and Rubén Cuevas. 2018. Unveiling and Quantifying Facebook Exploitation of Sensitive Personal Data for Advertising Purposes. In USENIX Security Symposium. 479--495. https://www.usenix.org/conference/usenixsecurity18/presentation/cabanas
- 10.4324/9781315582924--10 · doi:10.4324/9781315582924--10
- 10.1093/joc · doi:10.1093/joc
- Amit Datta, Anupam Datta, Jael Makagon, Deirdre K Mulligan, and Michael Carl Tschantz. 2018. Discrimination in Online Advertising: A Multidisciplinary Inquiry. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81 (2018), 20--34. http://proceedings.mlr.press/v81/datta18a/datta18a.pdf
- Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52---59. arxiv: 1906.02659 · arXiv
- Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. Digital Dominance: The Power of Google, Amazon, Facebook, and Apple (2018).
- Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
- Editors. 2020. Price Gouging Laws by State. https://consumer.findlaw.com/consumer-transactions/price-gouging-laws-by-state.html
- Elsevier. 2016. Scopus | The largest database of peer-reviewed literature.
- Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. "Be careful; Things can be worse than they appear" - Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 62--71. https://ojs.aaai.org/index.php/ICWSM/article/view/14898
- Anthony Flores, Christopher T. Lowenkamp, and Kristin Bechtel. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias". HeinOnline (2016).
- 10.7551/mitpress · doi:10.7551/mitpress
- 10.1093/bjsw · doi:10.1093/bjsw
- Shan Jiang, Ronald E. Robertson, and Christo Wilson. 2019 b. Bias misperceived: The role of partisanship and misinformation in YouTube comment moderation. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, Vol. 13(01). 278--289. https://ojs.aaai.org/index.php/ICWSM/article/view/3229
- Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. http://proceedings.mlr.press/v80/kearns18a.html
- Kyriakos Kyriakou, Pinar Barlas, Styliani Kleanthous, and Jahna Otterbacher. 2019. Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 313--322. https://www.aaai.org/ojs/index.php/ICWSM/article/view/3232/3100
- Emma Lurie and Eni Mustafaraj. 2019. Opening Up the Black Box: Auditing Google's Top Stories Algorithm. In Proceedings of the International Florida Artificial Intelligence Research Society Conference. https://par.nsf.gov/biblio/10101277
- Connor Mcmahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. In Proceedings of the 11th International AAAI Conference on Web and Social Media. https://ojs.aaai.org/index.php/ICWSM/article/view/14883
- 10.1145/2535372.2535415 · DOI · Paper page
- Safiya Noble. 2013. Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible Culture (2013).
- 10.1145/3292522.3326047 · DOI
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and discrimination: converting critical concerns into productive inquiry.
- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2020. The risk of racial bias in hate speech detection. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 1668--1678.
- 10.2139/ssrn.2208240 · DOI
- Nicholas Vincent, Isaac Johnson, Patrick Sheehan, and Brent Hecht. 2019 b. Measuring the Importance of User-Generated Content to Search Engines. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). https://ojs.aaai.org/index.php/ICWSM/article/view/3248
- Sara Wachter-Boettcher. 2017. Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sexism. WW Norton & Company.
Cited by
Items 101–196 of 196
Algorithmic Self-Diagnosis from Targeted Ads
Schoenebeck, S. Y., Goray, C., & Gunal, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710926 · Cited by 4
Panigutti, C., Yela, D. F., Porcaro, L., Bertrand, A., & Garrido, J. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732052 · Cited by 14
Opportunities and challenges of multidisciplinary algorithmic impact assessments
Sandoval, . C. B., Jing, F. S., Garcia, A. A., Berger, S., Candello, H. C. D. S. P., & Lustig, C. (2025) · Journal of Responsible Innovation · doi:10.1080/23299460.2025.2499302 · Cited by 6
Ren, B., Cheon, E., & Li, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757641 · Cited by 7
Heo, W. (2025) · Journal of Consumer Affairs · doi:10.1111/joca.70032 · Cited by 7
Tamer, O. Ü., McDonald, B. D., Hémici, F., & Κοντόγεωργα, Γ. (2025) · Sayıştay Dergisi · doi:10.52836/sayistay.1819606 · Cited by 3
Societal Implications of Recommendation Systems: A Technical Perspective
Gonçalves-Sá, J., & Pinheiro, F. L. (2023) · Law, governance and technology series · doi:10.1007/978-3-031-41264-6_3 · Cited by 8
Liao, J., Wei, Z., Yang, Z., Xu, X., Hui, P., He, C., & Zhou, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713763 · Cited by 6
Making Smart Cities Explainable: What XAI Can Learn from the “Ghost Map”
Gupta, S., & Loukissas, Y. A. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585847 · Cited by 4
Otterbacher, J. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3563359.3597381 · Cited by 5
Hacking Surveillance Cameras, Tricking AI and Disputing Biases: Artistic Critiques of Machine Vision
Kronman, L. (2023) · Open Library of Humanities · doi:10.16995/olh.10181 · Cited by 4
A Systematic Literature Review of Machine Learning Methods for Youtube Popularity Prediction
Yousif, H. H., & Mousa, A. H. (2026) · International Conference on Emerging Technologies and Engineering Systems (ICETES) · doi:10.1109/icetes68504.2026.11518668 · Cited by 2
Sued, G. (2021) · Anuario electrónico de estudios en Comunicación Social Disertaciones · doi:10.12804/revistas.urosario.edu.co/disertaciones/a.10616 · Cited by 4
Algorithmic bias: Looking beyond data bias to ensure algorithmic accountability and equity
Thais, S. J., Shumway, H., & Saragih, A. (2023) · MIT Science Policy Review · doi:10.38105/spr.5lwvw66ssy · Cited by 4
Under manipulations, are some AI models harder to audit?
Godinot, A., Merrer, E. L., Trédan, G., Penzo, C., & Taïani, F. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00038 · Cited by 11
Bias Reduction in Social Networks Through Agent-Based Simulations
Bartley, N., Burghardt, K., & Lerman, K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71975-2_5 · Cited by 2
Vincent, N., & Hecht, B. J. (2021) · Patterns · doi:10.1016/j.patter.2021.100388 · Cited by 2
Developing Responsible Algorithmic Curation Features in Social Media Through Participatory Design
Lee, K. S., & Wei, H. (2022) · [ ] With Design: Reinventing Design Modes · doi:10.1007/978-981-19-4472-7_188 · Cited by 5
Kazim, E., Polle, R., Carvalho, G., Koshiyama, A., Inness, C., Knight, A., Gorski, C., Barber, D., Lomas, E., Yılmaz, E., et al. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3935987 · Cited by 3
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AIPreprint
Deng, W. H., Wang, C., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Qeios · doi:10.32388/b1lxnv · Cited by 4
Global Approaches to Auditing Artificial Intelligence: A Literature Review
(IPIE), I. P. O. T. I. E., Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Сергеевич, Е. Р., Nelson, A., Sandvig, C., Sesan, G., et al. (2024) · doi:10.61452/bwym7397 · Cited by 7
Artificial Intelligence’s (AI’s) Responsible Use: How to Manage Digital Ethicswashing
Bernini, F., Ferretti, P., & Rosa, F. L. (2024) · SIDREA series in accounting and business administration · doi:10.1007/978-3-031-74292-7_2 · Cited by 1
Impacts of Personalization on Social Network Exposure
Bartley, N., Burghardt, K., & Lerman, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78538-2_3 · Cited by 4
The Influence of Social Media Algorithms on Small and Medium Enterprises in South Africa
Nkoana, I., & Mashamaite, K. A. (2025) · International Journal of Applied Research in Business and Management · doi:10.51137/wrp.ijarbm.2025.intt.45681 · Cited by 1
Paltaratskaya, V., Ji, A., Mazumdar, P., & Wise, K. R. (2025) · Journal of Current Issues & Research in Advertising · doi:10.1080/10641734.2025.2498996 · Cited by 2
The Problem of Ethics in Artificial Intelligence for Healthcare
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0002 · Cited by 1
P2NIA: Privacy-Preserving Non-iterative Auditing
Bourrée, J. G., Lautraite, H., Gambs, S., Trédan, G., Merrer, E. L., & Rottembourg, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06096-9_15 · Cited by 2
AI auditing through performance appraisals: A practice-informed approach
Kamphorst, B. A., Muis, I., Straatman, J., & Schäfer, M. T. (2025) · AI & Society · doi:10.1007/s00146-025-02674-3 · Cited by 2
Homophily-induced emergence of biased structures in LLM-based multi-agent AI systems
Mehdizadeh, A., & Hilbert, M. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01535-7 · Cited by 17
Audit-as-code: a policy-as-code framework for continuous AI assurance
Muhammad, A. E., Yow, K., & Alsenan, S. A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1759211 · Cited by 13
Algorithmic Audit of Personalisation Drift in Polarising Topics on TikTok
Pecher, B., Bindas, A., Jakubcik, J., Tuna, M., Tibenský, M., Liška, Š., Sakalik, P., Šutý, A., Mosnar, M., Hossner, F., et al. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3806161 · Cited by 3
A Method to Assess and Explain Disparate Impact in Online Retailing
Arreola, R. B. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583270 · Cited by 1
Granting Non-AI Experts Creative Control Over AI Systems
Lam, M. S. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3672539.3686714 · Cited by 1
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_2 · Cited by 1
Lassiter, T. B. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3682050 · Cited by 1
CLASSIFYING HUMANS: THE INDIRECT REVERSE OPERATIVITY OF MACHINE VISION
Kronman, L. (2023) · Photographies · doi:10.1080/17540763.2023.2189160 · Cited by 3
THE IMPACT OF INTERNET CONNECTIVITY IN NAVIGATING ONLINE SOCIAL NETWORKS: A CROSS-COUNTRY ANALYSISPreprint
Matlack, A. K., Boots, B. C., & Richardson-Gool, T. S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5208556 · Cited by 1
Patterns, Models, and Challenges in Online Social Media: A SurveyPreprint
Marco, A. D., Bonetti, A., Martino, E. D., Loru, E., Nudo, J., Pandolfo, M. E., Pecile, G., Sangiorgio, E., Scalco, I., Zollo, S., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5355426 · Cited by 1
Mökander, J. (2025) · Wiley eBooks · doi:10.1002/9781394240821.ch12 · Cited by 1
Measuring the Effectiveness of Ethics-Based Auditing – A Semi-Systematic Literature ReviewPreprint
Shafik, E., Puncheva, P., Bharathy, G., & Pannai, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.5498160 · Cited by 1
Auditing Machine Behaviors: Does a diverse crowd result in different social norms?
Christoforou, E., Orphanou, K., Nicolaou, N., Stavrakis, E., & Otterbacher, J. (2023) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc59560.2023.10386355
Bartolo, L. (2024) · Queensland University of Technology · doi:10.5204/thesis.eprints.247841 · Cited by 1
Çakmak, M. C., Agarwal, N., Okeke, O., Onyepunuka, U., & Spann, B. (2024) · Research Square · doi:10.21203/rs.3.rs-4421612/v1
Conclusions and Open Challenges
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_6
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_5
Biases, Fairness, and Non-discrimination
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_3 · Cited by 1
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_1
Schedl, M., Anelli, V. W., & Lex, E. (2024) · The information retrieval series · doi:10.1007/978-3-031-69978-8_4 · Cited by 1
The Rise of Social Media Platforms
Elizabeth, L. (2025) · Springer eBooks · doi:10.1007/978-3-031-87608-0_3 · Cited by 1
Facial Recognition Technologies: Threats or Opportunities for Democracy?
Mobilio, G. (2025) · Law, governance and technology series · doi:10.1007/978-3-031-89794-8_2 · Cited by 1
Designing Ethics into Infrastructures
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0004
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0005
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.002.0005
Introducing Smart Care, Dementia, and Care Ethics
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0001
Conclusion: Can Smart Care Be Ethical?
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0007
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.002.0004
Infrastructural Ethics of Smart Care
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0003
Lived Experience of Smart Care
Hine, C. (2025) · Oxford University Press eBooks · doi:10.1093/9780197764527.003.0006
Visualizing Children's Filter Bubbles: A Digital Window for Bridging Social Media Experiences
Bekavac, L. J. L., Bekavac, L., & Bekavac, B. (2025) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro65660.2025.11131921
Towards accurate context-aware user simulation for algorithm auditing
Zhao, Y., Han, H., Yang, W.-H., Zou, L., Liu, M., & Wang, L. (2025) · The Visual Computer · doi:10.1007/s00371-025-04160-9
Liu, Y., Cao, Q., Shen, H., Zhang, K., Wu, Y., & Cheng, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757414
Milea, T. D., & Siminiuc, R. (2025) · RoEduNet Conference: Networking in Education and Research (RoEduNet) · doi:10.1109/roedunet68395.2025.11208498 · Cited by 1
Damião, Í., Reis, J. M., Almeida, P., Santos, N., & Gonçalves-Sá, J. (2025) · IEEE Access · doi:10.1109/access.2025.3635871 · Cited by 3
Impact of Personalization Algorithms on News Reception
Makhortykh, M., & Vziatysheva, V. (2025) · Wiley eBooks · doi:10.1002/9781394250424.ch21
Paz, C. D. (2025) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas65609.2025.11269665
Der KI-basierte Rekrutierungsprozess im Personalwesen: Eine Literaturanalyse
Heinrich, K., & Parafeeva, E. (2026) · Springer eBooks · doi:10.1007/978-3-658-50111-2_12
377Rebuilding Knowledge: Epistemology and Knowledge Organization for Information Practice
Kelly, M. (2025) · De Gruyter eBooks · doi:10.1515/9783111346519-020
Pilania, U., Kumar, M., & Singh, S. (2026) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9798898813666126010014
Generality and explainability in recommender systems
Zhong, J. (2023) · theses.fr (ABES) · doi:10.70675/de5345e1z9ed3z4064zaaf4zc45a557c0632 · Cited by 2
Third Workshop on Human-Centered Evaluation and Auditing of Language Models: AI Agents-in-the-Loop
van der Maden, W., Deng, W. H., Liu, Y. L., Jiang, H., Chen, V., Li, H., Kim, J., Liao, Q. V., Xu, W., Eslami, M., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778709
Upadhyay, S., Pruss, D., DeVrio, A., Gajos, K. Z., & Karusala, N. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791923 · Cited by 2
Escher, N., Yrysov, B., McDermott, A., Chechelnitsky, D., Benyam, H. B., & Banović, N. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816926 · Cited by 1
PLATFORM GOVERNANCE AND ALGORITHMIC ACCOUNTABILITY IN SHAPING PUBLIC DISCOURSE
Sneh, A., Upadhyaya, D. B., Mishra, P., Kumar, S., & JAIN, M. (2026) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v7.i7s.2026.7652 · Cited by 1
Bandy, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-19438-1_2
Self-citationHumans as Observers: On the Reflexivity of Algorithm Auditing
束, 开., & 张, 昕. (2026) · Global Journal of Media Studies · doi:10.26599/gjms.2026.9330010
Gonzalez, D., Howley, N., Lurie, E., & Mustafaraj, E. (2026) · ACM Web Science Conference (WebSci) · doi:10.1145/3795513.3807436
Rohrbach, T., Makhortykh, M., & Sydorova, M. (2026) · Political Communication · doi:10.1080/10584609.2026.2679476 · Cited by 1
Fairness at a Glance: Can We Audit Model Fairness Before Training Completes?
Liu, Y., Cao, Q., Shen, H., & Cheng, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3794664
Shifting Audits Left: A Framework for Enhancing AI Auditability from an Internal Auditing Lens
Jamal, H., Faroukhi, A. Z., Elouataoui, W., Gharib, J., Gahi, Y., & El‐Khatib, K. (2026) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset68627.2026.11538661
Influência da Recomendação Algorítmica no Bem-Estar dos Usuários: Proposta de Auditoria
da Luz, A. B. F., & Figueiredo, M. C. (2026) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc_estendido.2026.20352
Everyday Auditing of TikTok's Generative AI Manga Filter
Encarnación, R., Morales‐Navarro, L., Kambhamettu, H., & Metaxa, D. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812379
Constraint-Aware Data Collection and Observational Analysis of Digital Labor Platforms
Frihaoui, A., Pons, O., & Lima, L. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-30863-4_12
Exposure to news and political content on TikTok: A Case Study Around the 2024 US Election
Ge, C., Gautam, N., Vajjhala, S., Ng, C., Hasell, A., & Tomkins, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812207 · Cited by 1
Auditing a Robotic System for the AI Act
Lucaj, L., Bok, F., & van der Smagt, P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812223 · Cited by 2
Algorithm Auditing Policies Rest on Flawed Assumptions About Public Sector Systems
Escher, N., Banović, N., & Green, B. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812280 · Cited by 2
Boban, M., Gombar, M., & Bachal, M. P. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591781
Paulke, H., Kruschinski, S., Bene, M., Haßler, J., Jackson, D., Lilleker, D. G., Magin, M., & Rußmann, U. (2026) · Social Science Computer Review · doi:10.1177/08944393261469406
Performing Rigor: Reflexivity in HCI/CSCW
McDonald, N., & Seberger, J. S. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3831595 · Cited by 3
Same System, Opposite Verdicts: Metric Discretion in AI Ethics Audits and the Limits of DisclosurePreprint
Tsaban, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10576395/v1
Iterative Netnography and Occurrence for Rapid Research: A Case Study of TikTok
Lutz, N., Hannuksela, K., Hynes, E., Le, C., Lopez, A., & Tomson, D. L. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816957
Cheng, C.-Y. (2026) · Electronic Commerce Research · doi:10.1007/s10660-026-10209-1
Algorithmic News Selection Under News Avoidance: Assessing News Values on Chinese Tiktok (Douyin)
Li, J., & Shi, W. (2026) · Journalism Studies · doi:10.1080/1461670x.2026.2739289
Trustworthy AI Is Not Valuable
Lim, J. (2026) · Journal of Applied Philosophy · doi:10.1002/japp.70137
Algorithmic Empowerment: The Fair Allocation of Visibility in Platformized Publics
Kwok, C. (2026) · Media Technology and Governance · doi:10.1177/29777984261494012
AI-Driven Auditing and Customer Experience in Global Audit Firms Towards Enhancing Digital Trust
Faisal, G. M., & Abbood, S. K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1508-7.ch002
Metadata
- Authors
- Bandy, J.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Bandy, Jack
- ORCID
- Bandy, Jack
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-34
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3159435787
- Cited by
- 193
- References
- 127
- Citations by year
- 2026: 31 · 2025: 60 · 2024: 45 · 2023: 32 · 2022: 18 · 2021: 7
- FWCI
- 31.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- algorithmic auditing, algorithmic discrimination, search algorithms, intersectional discrimination, audit procedures, algorithmic justice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.