The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Auditing Algorithms: Understanding Algorithmic Systems from the Outside In
Abstract
Algorithms are ubiquitous and critical sources of information online, increasingly acting as gatekeepers for users accessing or sharing information about virtually any topic, including their personal lives and those of friends and family, news and politics, entertainment, and even information about health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. However, studying such content poses considerable challenges, as it is both dynamic and ephemeral: these algorithms are constantly changing, and frequently changing silently, with no record of the content to which users have been exposed over time. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm’s opaque inner workings and possible external impact. In this work, we present an overview of the algorithm audit methodology, including the history of audit studies in the social sciences from which this method is derived; a summary of key algorithm audits over the last two decades in a variety of domains, including health, politics, discrimination, and others; and a set of best practices for conducting algorithm audits today, contextualizing these practices using search engine audits as a case study. Finally, we conclude by discussing the social, ethical, and political dimensions of auditing algorithms, and propose normative standards for the use of this method.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our study audits two commercially deployed systems across four genres under specific prompting regimes, but does not identify causal mechanisms behind the observed patterns, a common constraint of black-box audits (Metaxa et al. 2021).”
From The Algorithmic Flattening of Sound: Computational Evidence and Justice Implications of AI Music Homogenization (2026) · cites this paper as Metaxa et al. 2021
Citations
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 32,792
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Getting a Job: Is There a Motherhood Penalty?
Correll, S. J., Benard, S., & Paik, I. (2007) · American Journal of Sociology · doi:10.1086/511799 · Cited by 2,525
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Cyberchondria: studies of the escalation of medical concerns in web search
White, R. W., & Horvitz, E. (2009) · ACM Transactions on Information Systems · doi:10.1145/1629096.1629101 · Cited by 546
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
Programmed Visions: Software and Memory
Chun, W. H. K. (2011) · doi:10.7551/mitpress/9780262015424.001.0001 · Cited by 475
Nosek, B. A., & Errington, T. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000691 · Cited by 444
The Googlization of Everything: (And Why We Should Worry)
Vaidhyanathan, S. (2011) · doi:10.1525/9780520948693 · Cited by 362
Temporal Distance and Discrimination: An Audit Study in Academia
Milkman, K. L., Akinola, M., & Chugh, D. (2012) · Psychological Science · doi:10.1177/0956797611434539 · Cited by 267
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
Advances in nowcasting influenza-like illness rates using search query logs
Lampos, V., Miller, A. C., Crossan, S., & Stefansen, C. (2015) · Scientific Reports · doi:10.1038/srep12760 · Cited by 143
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
Self-citationMowshowitz, A., & Kawaguchi, A. (2005) · Information Processing & Management · doi:10.1016/j.ipm.2004.05.005 · Cited by 68
How search engines disseminate information about COVID-19 and why they should do better
Makhortykh, M., Urman, A., & Ulloa, R. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-017 · Cited by 54
Research works printed but not linked: 89
- Abebe R. Hill S. Vaughan J. W. Small P. M. Schwartz H. A. 2019 “Using Search Queries to Understand Health Information Needs in Africa” Proceedings of the International AAAI Conference on Web and Social Media 13 01 3–14https://ojs.aaai.org/index.php/ICWSM/article/view/3360 · DOI
- Abid, (2021), “Large language models associate Muslims with violence”, Nature Machine Intelligence, 3, 461, 10.1038/s42256-021-00359-2 · DOI
- ACLU (2020) “Federal Court Rules ‘Big Data’ Discrimination Studies Do Not Violate Federal Anti-Hacking Law” Mar. https://www.aclu.org/press-releases/federal-court-rules-big-data-discrimination-studies-do-not-violate-federal-anti (accessed on 08/26/2020)
- Allam, (2014), “The Impact of Search Engine Selection and Sorting Criteria on Vaccination Beliefs and Attitudes:Two Experiments Manipulating Google Output”, Journal of Medical Internet Research, 16, 10.2196/jmir.2642 · DOI
- Asplund, 2020, “Auditing Race and Gender Discrimination in Online Housing Markets”, Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2020), 10
- Asplund, 2020, “Auditing race and gender discrimination in online housing markets” In:, Proceedings of the International AAAI Conference on Web and Social Media, 14, 24, 10.1609/icwsm.v14i1.7276 · DOI
- Ballatore, (2015), “Google Chemtrails: A Methodology to Analyze Topic Representation in Search Engine Results”, First Monday, 20, 10.5210/fm.v20i7.5597 · DOI
- Ballatore, (2017), “Digital Hegemonies: The Localness of Search Engine ”, Annals of the American Association of Geographers, 107, 1194, 10.1080/24694452.2017.1308240 · DOI
- Bandy J. (2021) “Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits” arXiv:2102.04256 [cs.CY] · DOI · Paper page
- Bandy, 2020, “Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News”, Proceedings of the International AAAI Conference on Web and Social Media, 14, 36, 10.1609/icwsm.v14i1.7277 · DOI
- Barocas, 2017, “Fairness in machine ”, NIPS Tutorial, 1
- Bechmann, 2018, “Are We Exposed to the Same “News” in the News Feed?”, Digital Journalism, 6, 990, 10.1080/21670811.2018.1510741 · DOI
- Bernstein, 2021, “ESR: Ethics and Society Review of Artificial Intelligence Research”
- Bobrowsky, 2021, “Facebook Disables Access for NYU Research Into Political-Ad Targeting”, The Wall Street Journal
- Brown, 2021, “The algorithm audit: Scoring the algorithms that score us”, Big Data & Society, 8, 10.1177/2053951720983865 · DOI · Paper page
- Buolamwini, 2018, “Gender shades: Intersectional accuracy disparities in commercial gender classification” In:, Conference on Fairness, Accountability and Transparency, 77
- Cartright, 2011, “Intentions and Attention in Exploratory Health Search” In:, Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, 65
- Chen, 2018, “Investigating the Impact of Gender on Rank in Resume Search Engines” In:, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 1, 10.1145/3173574.3174225 · DOI · Paper page
- Chen, 2015, “Peeking beneath the hood of Uber” In:, Proceedings of the 2015 Internet Measurement Conference, 495, 10.1145/2815675.2815681 · DOI · Paper page
- Cobbe, 2021, “Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems”, Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21), 598, 10.1145/3442188.3445921 · DOI · Paper page
- Association for Computing Machinery 1992 “ACM Code of Ethics and Professional Conduct” Code of Ethics
- Daniel, 1968, “Racial discrimination in England: based on the PEP report”, 1084
- DeChoudhury, 2014, “Seeking and Sharing Health Information Online: Comparing Search Engines and Social Media”, Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, 1365
- Diakopoulos, 2018, “I vote for—how search informs our choice of candidate”, Digital Dominance: The Power of Google, Amazon, Facebook, and Apple, 22
- Díaz, 2018, “Addressing age-related bias in sentiment analysis”In:, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1
- Edelman, 2017, “Racial discrimination in the sharing economy: Evidence from a field ”, American Economic Journal: Applied Economics, 9, 1
- Edelman B. G. Luca M. 2014 “Digital discrimination: The case of Airbnb.com” Harvard Business School NOM Unit Working Paper 14 54 · DOI · Paper page
- Epstein, 2015, “The Search Engine Manipulation Effect (SEME) and Its Possible Impact on the Outcomes of Elections”, Proceedings of the National Academy of Sciences, 112, E4512, 10.1073/pnas.1419828112 · DOI · Paper page
- Epstein, 2017, “Suppressing the Search Engine Manipulation Effect (SEME)”, Proceedings of the ACM on Human-Computer Interaction, 1, 1, 10.1145/3134677 · DOI
- Eriksson, 2017, “Tracking gendered streams”, Culture Unbound, 9, 163, 10.3384/cu.2000.1525.1792163 · DOI
- Fortunato, 2006, “Topical Interests and the Mitigation of Search Engine Bias”, Proceedings of the National Academy of Sciences, 103, 12684, 10.1073/pnas.0605525103 · DOI
- Gaddis, 2018, Audit studies: Behind the scenes with theory, method, and nuance, 10.1007/978-3-319-71153-9 · DOI · Paper page
- Ginsberg, 2009, “Detecting Influenza Epidemics Using Search Engine Query dataprivacylab”, Nature, 457, 1012, 10.1038/nature07634 · DOI · Paper page
- Gneezy, 2004, “the Are the disabled discriminated against in product markets? Evidence from field experiments”, American Economic Association Annual Meeting
- Goldman, 2008, Web Search, 121, 10.1007/978-3-540-75829-7_8 · DOI · Paper page
- Granka, 2010, The“ Politics of Search: A Decade Retrospective”, The Information Society, 26, 364, 10.1080/01972243.2010.511560 · DOI
- Hannak, 2013, “Measuring Personalization of Web Search”, Proceedings of the 22nd International Conference on World Wide Web (WWW '13), 527, 10.1145/2488388.2488435 · DOI · Paper page
- Hannak, 2014, “Measuring price discrimination and steering on e-commerce web sites”, Proceedings of the 2014 Internet Measurement Conference, 305, 10.1145/2663716.2663744 · DOI · Paper page
- Hannak, 2017, “Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr”, 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017), 10.1145/2998181.2998327 · DOI · Paper page
- Hu, 2019, “Auditing the Partisanship of Google Search Snippets”, The Web Conference (WWW ’19), 693, 10.1145/3308558.3313654 · DOI
- Hussein, 2020, “Measuring Misinformation in Video Search Platforms: An Audit Study on YouTube”, Proceedings of the ACM on Human-Computer Interaction, 4, 1, 10.1145/3392854 · DOI · Paper page
- Hutchinson, 2019, “50 Years of Test (Un)Fairness: Lessons for Machine Learning”, Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19), 49, 10.1145/3287560.3287600 · DOI · Paper page
- Introna, 2000, “Shaping the Web: Why the Politics of Search Engines Matters”, The Information Society, 16, 169, 10.1080/01972240050133634 · DOI
- Joachims, 2007, “Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search”, ACM Transactions on Information Systems, 25, 7, 10.1145/1229179.1229181 · DOI · Paper page
- Juneja, 2021, Auditing“ E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation”, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21), 10.1145/3411764.3445250 · DOI
- Kawakami, 2020, “The Media Coverage of the 2020 US Presidential Election Candidates through the Lens of Google’s Top Stories”, Proceedings of the International AAAI Conference on Web and Social Media, 14, 868, 10.1609/icwsm.v14i1.7352 · DOI
- Kay, 2015, “Unequal Representation and Gender Stereotypes in Image Search Results for Occupations”, Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 3819, 10.1145/2702123.2702520 · DOI · Paper page
- Keyes, 2019, “A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the Elderly into High-Nutrient Slurry”, Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19), 10.1145/3290607.3310433 · DOI
- Kliman-Silver, 2015, “Location, Location, Location: The Impact of Geolocation on Web Search Personalization”, Proceedings of the 2015 ACM Internet Measurement Conference (IMC '15), 121, 10.1145/2815675.2815714 · DOI
- Kugelmass, 2016, ““Sorry, I'm Not Accepting New Patients”: An Audit Study of Access to Mental Health Care”, Journal of Health and Social Behavior, 57, 168, 10.1177/0022146516647098 · DOI
- Kulshrestha, 2019, Search Bias Quantification: Investigating Political Bias in Social Media and Web Search, Information Retrieval Journal, 22, 188, 10.1007/s10791-018-9341-2 · DOI
- Lawrence, 1999, “Accessibility of Information on the Web”, Nature, 400, 107, 10.1038/21987 · DOI · Paper page
- Lazer, 2014, “The Parable of Google Flu: Traps in Big Data Analysis”, Science, 343, 1203, 10.1126/science.1248506 · DOI · Paper page
- Le, 2019, “Measuring Political Personalization of Google News Search”, The World Wide Web Conference (WWW ’19), 2957, 10.1145/3308558.3313682 · DOI
- Lurie, 2018, “ Investigating the Effects of Google’s Search Engine Result Page in Evaluating the Credibility of Online News Sources”, Proceedings of the 10th ACM Conference on Web Science (WebSci '18), 107, 10.1145/3201064.3201095 · DOI
- McCown, 2007, –Agreeing to Disagree: Search Engines and Their Public Interfaces–, Proceedings of the 2007 Conference on Digital Libraries (JCDL'07), 309, 10.1145/1255175.1255237 · DOI
- McMahon, 2017, –The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship between Peer Production Communities and Information Technologies–, Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2018), 10
- Mehrotra, 2017, –Auditing Search Engines for Differential Satisfaction Across Demographics–, Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion), 626, 10.1145/3041021.3054197 · DOI
- Metaxa, 2019, –Search Media and Elections: A Longitudinal Investigation of Political Search Results in the 2018 U.S. Elections–, Proceedings of the 22nd ACM Conference on Computer-Supported Cooperative Work and Social Computing
- Metaxas, 2009, –The Battle for the 2008 US Congressional Elections on the Web–, Proceedings of the 2009 WebScience: Society On-Line Conference
- Metaxas, 2017, –Manipulation of Search Engine Results during the 2016 US Congressional Elections–, Proceedings of the ICIW 2017, 6
- Mikians, 2012, –Detecting price and search discrimination on the internet–, Proceedings of the 11th ACM Workshop on Hot Topics in Networks, 79
- Milan, 2016, "The alternative epistemologies of data activism", Digital Culture & Society, 2, 57, 10.14361/dcs-2016-0205 · DOI · Paper page
- Mowshowitz, 2002, "Assessing Bias in Search Engines, Information Processing & Management", 38, 141, 10.1016/S0306-4573(01)00020-6 · DOI
- Mowshowitz, 2002, "Bias on the Web", Communications of the ACM, 45, 10.1145/567498.567527 · DOI
- Mullainathan S. Noeth M. Schoar A. 2012 "The market for financial advice: An audit study" National Bureau of Economic Research Tech. Report · DOI · Paper page
- Mustafaraj, 2020, "The Case for Voter-Centered Audits of Search Engines During Political Elections", Conference on Fairness, Accountability, and Transparency (FAT* ’20), 11
- Noble, 2013, "Google search: Hyper-visibility as a means of rendering black women and girls invisible", InVisible Culture
- Noble, 2018, "Google Has a Striking History of Bias Against Black Girls", Time Magazine
- Obermeyer, 2019, "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations", Science, 366, 447, 10.1126/science.aax2342 · DOI · Paper page
- Oreopoulos, 2011, "Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes", American Economic Journal: Economic Policy, 3, 148
- Pan, 2007, "In Google We Trust: Users’ Decisions on Rank, Position, and Relevance", Journal of Computer-Mediated Communication, 12, 801, 10.1111/j.1083-6101.2007.00351.x · DOI · Paper page
- Pitoura, 2018, "On Measuring Bias in Online Information", ACM SIGMOD Record, 46, 16, 10.1145/3186549.3186553 · DOI
- Prosperi, 2019, "Is it time to rethink institutional review boards for the era of big data?", Nature Machine Intelligence, 1, 260, 10.1038/s42256-019-0059-7 · doi:10.1038/s42256-019-0059-7
- Robertson, 2018, "Auditing Partisan Audience Bias within Google Search", Proceedings of the ACM on Human-Computer Interaction, 2, 1, 10.1145/3274417 · DOI · Paper page
- Robertson, 2019, "Auditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation", Proceedings of the 10th ACM Conference on Web Science (WebSci '19), 235, 10.1145/3292522.3326047 · DOI
- Robertson, 2018, "Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages", 955, 10.1145/3178876.3186143 · DOI
- Rothshild, 2019, "How the Interplay of Google and Wikipedia Affects Perceptions of Online News Sources"
- Sapiezynski, 2019, "Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists", 553, 10.1145/3308560.3317595 · DOI · Paper page
- Sharma, 2015, "Insurance, race/ethnicity, and sex in the search for a new physician", Economics Letters, 137, 150, 10.1016/j.econlet.2015.11.005 · DOI
- Speicher, 2018, "Potential for discrimination in online targeted advertising", Conference on Fairness, Accountability, and Transparency (FAT '20)
- Sweeney L. 2013 "Discrimination in Online Ad Delivery" https://dataprivacylab.org/projects/onlineads/index.html Accessed 10/26/2020 · DOI
- "Google’s Algorithm – Seek and You Shall Find", 2019, The Economist, 81
- Trielli, 2019, "Search as News Curator: The Role of Google in Shaping Attention to News Information", Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 1, 10.1145/3290605.3300683 · DOI
- Trielli, 2020, "Partisan Search Behavior and Google Results in the 2018 U.S. Midterm Elections", Information, Communication & Society, 1, 10.1080/1369118X.2020.1764605 · DOI
- Vincent, 2021, "A Deeper Investigation of the Importance of Wikipedia Links to Search Engine Results", Proceedings of the ACM on Human-Computer Interaction, 5, 10.1145/3449078 · DOI
- Vincent, 2019, "Measuring the Importance of User-Generated Content to Search Engines", Proceedings of the International AAAI Conference on Web and Social Media, 13, 10.1609/icwsm.v13i01.3248 · DOI
- Xing, 2014, "Exposing Inconsistent Web Search Results with Bobble", Passive and Active Measurement, 131, 10.1007/978-3-319-04918-2_13 · DOI
- Yinger, 1998, "Evidence on discrimination in consumer markets", Journal of Economic Perspectives, 12, 23, 10.1257/jep.12.2.23 · DOI · Paper page
Cited by
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“(Metaxa et al., 2021, p.18) Narrow conceptions of auditing are well suited to gathering evidence about unlawful discrimination and tend to be underpinned by experimental designs.”
BackgroundDeng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
Ojewale, V., Steed, R., Vecchione, B., Birhane, A., & Raji, I. D. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713301 · Cited by 81
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
Metadata
- Authors
- Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C.
- Affiliations
- Stanford University
- Northeastern University
- University of Illinois at Urbana-Champaign
- University of Michigan
- Venue
- Foundations and Trends® in Human–Computer Interaction
- Publisher
- Emerald
- Published
- 2021-11-25
- Volume
- 14
- Issue
- 4
- Pages
- 272-344
- ISSN
- 1551-3955
- 1551-3963
- Cited by
- 140
- References
- 15
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.