CoChat
Sign Up
Cited by 8

The right to audit and power asymmetries in algorithm auditing

Urman, Aleksandra, Smirnov, Ivan, Lasser, Jana·2024·EPJ Data Science
DOI 10.1140/epjds/s13688-024-00454-5
Open this paper in CoChat — free account

Abstract

In this paper, we engage with and expand on the keynote talk about the “Right to Audit” given by Prof. Christian Sandvig at the International Conference on Computational Social Science 2021 through a critical reflection on power asymmetries in the algorithm auditing field. We elaborate on the challenges and asymmetries mentioned by Sandvig — such as those related to legal issues and the disparity between early-career and senior researchers. We also contribute a discussion of the asymmetries that were not covered by Sandvig but that we find critically important: those related to other disparities between researchers, incentive structures related to the access to data from companies, targets of auditing and users and their rights. We also discuss the implications these asymmetries have for algorithm auditing research such as the Western-centrism and the lack of the diversity of perspectives. While we focus on the field of algorithm auditing specifically, we suggest some of the discussed asymmetries affect Computational Social Science more generally and need to be reflected on and addressed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1140/epjds/s13688-024-00454-5

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • External auditors face the threat of legal action under existing privacy and anti-hacking laws.

    “who face the threat of legal action under existing privacy and anti-hacking laws [88, 122]”

    From Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling (2025) · cites this paper as [122]

    Section 4 reports threats of legal action against independent auditors, and the Introduction discusses a challenge to the Computer Fraud and Abuse Act. The paper does not establish that auditors face legal action under privacy laws; its discussion of privacy concerns does not make that connection. No page numbers were provided in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Hannak A et al (2013) Measuring personalization of web search. In: Proceedings of the 22nd international conference on world wide web, New York, NY, USA, pp 527-538. https://doi.org/10.1145/2488388.2488435 · DOI · Paper page
  2. Gaddis SM (2018) An introduction to audit studies in the social sciences. In: Gaddis SM (ed) Audit studies: behind the scenes with theory, method, and nuance. Springer, Cham, pp 3-44. https://doi.org/10.1007/978-3-319-71153-9_1 · DOI · Paper page
  3. Johnson DA, Porter RJ, Mateljan PL (1971) Racial discrimination in apartment rentals. J Appl Soc Psychol 1(4):364-377. https://doi.org/10.1111/j.1559-1816.1971.tb00373.x · DOI
  4. Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J (2012) Science faculty's subtle gender biases favor male students. Proc Natl Acad Sci 109(41):16474-16479. https://doi.org/10.1073/pnas.1211286109 · DOI · Paper page
  5. Bessudnov A, Shcherbak A (2020) Ethnic discrimination in multi-ethnic societies: evidence from Russia. Eur Sociol Rev 36(1):104-120. https://doi.org/10.1093/esr/jcz045 · DOI
  6. Pager D (2007) The use of field experiments for studies of employment discrimination: contributions, critiques, and directions for the future. Ann Am Acad Polit Soc Sci 609(1):104-133. https://doi.org/10.1177/0002716206294796 · DOI
  7. Boeker M, Urman A (2022) An empirical investigation of personalization factors on TikTok. In: Proceedings of the ACM web conference 2022, New York, NY, USA, pp 2298-2309. https://doi.org/10.1145/3485447.3512102 · DOI
  8. Scherr S, Arendt F, Haim M (2023) Algorithms without frontiers? How language-based algorithmic information disparities for suicide crisis information sustain digital divides over time in 17 countries. Inf Commun Soc 26(14):2690-2706. https://doi.org/10.1080/1369118X.2022.2097017 · DOI
  9. Dambanemuya HK, Diakopoulos N (2021) Auditing the information quality of news-related queries on the Alexa voice assistant. Proc ACM Hum-Comput Interact 5(CSCW1):83:1-83:21. https://doi.org/10.1145/3449157 · DOI
  10. Bandy J, Diakopoulos N (2021) Curating quality? How Twitter's timeline algorithm treats different types of news. Soc Media Soc 7(3):20563051211041650. https://doi.org/10.1177/20563051211041648 · DOI
  11. Noble SU (2018) Algorithms of oppression: how search engines reinforce racism. NYU Press, New York
  12. Urman A, Makhortykh M, Ulloa R, Kulshrestha J (2022) Where the Earth is flat and 9/11 is an inside job: a comparative algorithm audit of conspiratorial information in web search results. Telemat Inform 72:101860. https://doi.org/10.1016/j.tele.2022.101860 · DOI
  13. Ugwudike P (2022) AI audits for assessing design logics and building ethical systems: the case of predictive policing algorithms. AI Ethics 2(1):199-208. https://doi.org/10.1007/s43681-021-00117-5 · DOI
  14. Mark F (2020) The sociology of debt. Policy Press, Bristol
  15. Mund BZ (2018) Protecting deceptive academic research under the computer fraud and abuse act. Yale Law Policy Rev 37(1):385-403
  16. Patel KS. Testing the limits of the first amendment: how online civil rights testing is protected speech activity. Columbia Law Review. https://columbialawreview.org/content/testing-the-limits-of-the-first-amendment-how-online-civil-rights-testing- is-protected-speech-activity/. Accessed 06 Feb 2024
  17. Laupman C, Schippers L-M, Papaléo Gagliardi M (2022) Biased algorithms and the discrimination upon immigration policy. In: Custers B, Fosch-Villaronga E (eds) Law and artificial intelligence: regulating AI and applying AI in legal practice. T.M.C. Asser Press, The Hague, pp 187-204. https://doi.org/10.1007/978-94-6265-523-2_10 · DOI
  18. ResearchGate. Terms of service. ResearchGate. https://www.researchgate.net/terms-of-service. Accessed 24 Nov 2022
  19. Weisshaar K (2017) Publish and perish? An assessment of gender gaps in promotion to tenure in academia. Soc Forces 96(2):529-560. https://doi.org/10.1093/sf/sox052 · DOI
  20. Chen J, Liu Q, Kim M (2022) Gender gap in tenure and promotion: evidence from the economics Ph.D. class of 2008. South Econ J 88(4):1277-1312. https://doi.org/10.1002/soej.12567 · DOI
  21. Steinþórsdóttir FS, Einarsdóttir Þ, Pétursdóttir GM, Himmelweit S (2020) Gendered inequalities in competitive grant funding: an overlooked dimension of gendered power relations in academia. High Educ Res Dev 39(2):362-375. https://doi.org/10.1080/07294360.2019.1666257 · DOI
  22. Wagner MW (2023) Independence by permission. Science 381(6656):388-391. https://doi.org/10.1126/science.adi2430 · DOI
  23. Puschmann C (2019) Beyond the bubble: assessing the diversity of political search results. Dig Journal 7(6):824-843. https://doi.org/10.1080/21670811.2018.1539626 · DOI
  24. Haim M, Graefe A, Brosius H-B (2018) Burst of the filter bubble?: effects of personalization on the diversity of Google news. Dig Journal 6(3):330-343. https://doi.org/10.1080/21670811.2017.1338145 · DOI · Paper page
  25. Unkel J, Haim M (2021) Googling politics: parties, sources, and issue ownerships on Google in the 2017 German federal election campaign. Soc Sci Comput Rev 39(5):844-861. https://doi.org/10.1177/0894439319881634 · DOI
  26. Makhortykh M, Urman A, Ulloa R (2021) Hey, Google, is it what the Holocaust looked like?: auditing algorithmic curation of visual historical content on web search engines. First Monday. https://doi.org/10.5210/fm.v26i10.11562 · DOI
  27. Makhortykh M, Urman A, Wijermars M (2022) A story of (non)compliance, bias, and conspiracies: how Google and Yandex represented Smart Voting during the 2021 parliamentary elections in Russia. Harv Kennedy Sch Misinformation Rev. https://doi.org/10.37016/mr-2020-94 · DOI
  28. Makhortykh M, Urman A, Ulloa R (2022) This is what a pandemic looks like: visual framing of COVID-19 on search engines. https://doi.org/10.48550/arXiv.2209.11120 · DOI
  29. Jiang M (2014) The business and politics of search engines: a comparative study of Baidu and Google's search results of Internet events in China. New Media Soc 16(2):212-233. https://doi.org/10.1177/1461444813481196 · DOI
  30. Kravets D, Toepfl F (2022) Gauging reference and source bias over time: how Russia's partially state-controlled search engine Yandex mediated an anti-regime protest event. Inf Commun Soc 25(15):2207-2223. https://doi.org/10.1080/1369118X.2021.1933563 · DOI
  31. Toepfl F, Kravets D, Ryzhova A, Beseler A (2023) Who are the plotters behind the pandemic? Comparing Covid-19 conspiracy theories in Google search results across five key target countries of Russia's foreign communication. Inf Commun Soc 26(10):2033-2051. https://doi.org/10.1080/1369118X.2022.2065213 · DOI
  32. Zavadski A, Toepfl F (2019) Querying the Internet as a mnemonic practice: how search engines mediate four types of past events in Russia. Media Cult Soc 41(1):21-37. https://doi.org/10.1177/0163443718764565 · DOI
  33. Haim M (2020) Agent-based testing: an automated approach toward artificial reactions to human behavior. J Stud 21(7):895-911. https://doi.org/10.1080/1461670X.2019.1702892 · DOI
  34. Ulloa R, Makhortykh M, Urman A (2022) Scaling up search engine audits: practical insights for algorithm auditing. J Inf Sci, 01655515221093029. https://doi.org/10.1177/01655515221093029 · DOI

Cited by

Sorted by citations

Authors
Urman, A., Smirnov, I., & Lasser, J.
ORCID
Urman, Aleksandra
Venue
EPJ Data Science
Publisher
Springer Science and Business Media LLC
Published
2024-03-07
Volume
13
Issue
1
ISSN
2193-1127
DOI
10.1140/epjds/s13688-024-00454-5
Cited by
8
References
7
Language
en
Licence
Funders
H2020 Marie Skłodowska-Curie Actions (101026507)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review