CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

The Surveillance AI Pipeline

Kalluri, Pratyusha Ria, Agnew, William, Cheng, Myra, Owens, Kentrell, Soldaini, Luca, Birhane, Abeba·2023·arXiv
DOI 10.48550/arxiv.2309.15084arXiv 2309.15084
Open this paper in CoChat — free account

Abstract

A rapidly growing number of voices argue that AI research, and computer vision in particular, is powering mass surveillance. Yet the direct path from computer vision research to surveillance has remained obscured and difficult to assess. Here, we reveal the Surveillance AI pipeline by analyzing three decades of computer vision research papers and downstream patents, more than 40,000 documents. We find the large majority of annotated computer vision papers and patents self-report their technology enables extracting data about humans. Moreover, the majority of these technologies specifically enable extracting data about human bodies and body parts. We present both quantitative and rich qualitative analysis illuminating these practices of human data extraction. Studying the roots of this pipeline, we find that institutions that prolifically produce computer vision research, namely elite universities and "big tech" corporations, are subsequently cited in thousands of surveillance patents. Further, we find consistent evidence against the narrative that only these few rogue entities are contributing to surveillance. Rather, we expose the fieldwide norm that when an institution, nation, or subfield authors computer vision papers with downstream patents, the majority of these papers are used in surveillance patents. In total, we find the number of papers with downstream surveillance patents increased more than five-fold between the 1990s and the 2010s, with computer vision research now having been used in more than 11,000 surveillance patents. Finally, in addition to the high levels of surveillance we find documented in computer vision papers and patents, we unearth pervasive patterns of documents using language that obfuscates the extent of surveillance. Our analysis reveals the pipeline by which computer vision research has powered the ongoing expansion of surveillance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.15084

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern machine learning techniques afford potential for mass surveillance.

    “Of particular relevance is work which examines the potential for mass surveillance which modern machine learning techniques afford and which widespread agents with episodic memories might contribute to [60].”

    From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

    Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F., Cheng, M., Nozza, D., Hashimoto, T., Jurafsky, D., Zou, J., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594095 · Cited by 342 · cited 1×

    “…rise and accessibility of generative models such as Stable Diffusion, not only exacerbates the erosion of privacy and the reproduction of social stereotypes, toxic and discriminatory predictions, the proliferation of these generated images at a massive scale also pollutes the digital ecology [16].”
    BackgroundSelf-citation

Flagged first, then by citations

Research works printed but not linked: 47

  1. 2021. CVPR 2021 Report Identifies 5 Trend Areas. https://www.computer.org/publications/tech-news/events/cvpr-2021-recap.
  2. Janet Abbate. 2000. Inventing the internet. MIT press.
  3. Janet Abbate. 2012. Recoding gender: Women's changing participation in computing. Mit Press.
  4. Philip E Agre. 1994. Surveillance and capture: Two models of privacy. The information society 10, 2 (1994), 101-127.
  5. Mohammad Ahmadpoor and Benjamin F. Jones. 2017. The dual frontier: Patented inventions and prior scientific advance. Science 357, 6351 (2017), 583-587. https://doi.org/10.1126/science.aam9527 arXiv:https://www.science.org/doi/pdf/10.1126/science.aam9527 · DOI
  6. Nur Ahmed and Muntasir Wahed. 2020. The De-democratization of AI: Deep Learning and the Compute Divide in Artificiatl Intelligence Research. arXiv (2020).
  7. Thomas Allmer. 2011. Critical surveillance studies in the information society. tripleC: Communication, Capitalism & Critique. Open Access Journal for a Global Sustainable Information Society 9, 2 (2011), 566-592. https://www.triple-c.at/index.php/tripleC/article/view/266
  8. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The values encoded in machine learning research. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 173-184.
  9. Abeba Birhane and Vinay Uday Prabhu. 2021. Large image datasets: A pyrrhic win for computer vision?. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1536-1546.
  10. Sarah R Blenner, Melanie Köllmer, Adam J Rouse, Nadia Daneshvar, Curry Williams, and Lori B Andrews. 2016. Privacy policies of android diabetes apps and sharing of health information. Jama 315, 10 (2016), 1051-1052.
  11. Meredith Broussard. 2018. Artificial unintelligence: How computers misunderstand the world. mit Press.
  12. Simone Browne. 2015. Dark matters: On the surveillance of blackness. Duke University Press.
  13. Julie E Cohen. 2017. Surveillance vs. privacy: effects and implications. Cambridge Handbook of Surveillance Law, eds.
  14. David Gray & Stephen E. Henderson (New York: Cambridge University Press, 2017) (2017), 455-69.
  15. Kate Conger, Richard Fausset, and Serge F Kovaleski. 2019. San Francisco bans facial recognition technology. The New York Times 14 (2019), 1.
  16. Kate Crawford. 2021. The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
  17. Lydia de la Torre. 2018. A guide to the california consumer privacy act of 2018. Available at SSRN 3275571 (2018).
  18. Gilles Deleuze. 1992. Postscript on the Societies of Control. The MIT Press.
  19. Paul N Edwards. 1996. The closed world: Computers and the politics of discourse in Cold War America. MIT press.
  20. Satu Elo and Helvi Kyngäs. 2008. The qualitative content analysis process. Journal of advanced nursing 62, 1 (2008), 107-115.
  21. Nathan Ensmenger. 2015. "Beards, sandals, and other signs of rugged individualism": masculine culture within the computing professions. Osiris 30, 1 (2015), 38-65.
  22. Nathan L Ensmenger. 2012. The computer boys take over: Computers, programmers, and the politics of technical expertise. Mit Press.
  23. Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.
  24. Steven Feldstein. 2019. The global expansion of AI surveillance. Vol. 17. Carnegie Endowment for International Peace Washington, DC.
  25. Björn Hammarfelt. 2021. Linking science to technology: the "patent paper citation. Journal of Documentation (2021).
  26. Donna Haraway. 2020. Situated knowledges: The science question in feminism and the privilege of partial perspective. In Feminist theory reader. Routledge, 303-310.
  27. Sandra Harding. 2013. Rethinking standpoint epistemology: What is "strong objectivity"? In Feminist epistemologies. Routledge, 49-82.
  28. Melvyn P Leffler and Odd Arne Westad. 2010. The Cambridge history of the cold war. Vol. 1. Cambridge University Press.
  29. Stuart W Leslie et al. 1993. The Cold War and American science: The military-industrial-academic complex at MIT and Stanford. Columbia University Press.
  30. Helen E Longino. 2020. Science as social knowledge. In Science as Social Knowledge. Princeton university press.
  31. Gianclaudio Malgieri. 2020. The concept of fairness in the GDPR: a linguistic and contextual interpretation. In Proceedings of the 2020 Conference on fairness, accountability, and transparency. 154-166.
  32. Gary T. Marx. 2015. Surveillance Studies. In International Encyclopedia of the Social & Behavioral Sciences (Second Edition) (second edition ed.), James D. Wright (Ed.). Elsevier, Oxford, 733-741. https://doi.org/10.1016/B978-0-08-097086-8.64025-4 · DOI
  33. Matt Marx and Aaron Fuegi. 2022. Reliance on science by inventors: Hybrid extraction of in-text patent-to-article citations. Journal of Economics & Management Strategy 31, 2 (2022), 369-392.
  34. Torin Monahan and David Murakami Wood. 2018. Surveillance Studies: A Reader. Oxford University Press.
  35. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin'words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis 16, 4 (2008), 372-403.
  36. Irena Nesterova. 2022. Questioning the EU proposal for an Artificial Intelligence Act: The need for prohibitions and a stricter approach to biometric surveillance. Information Polity Preprint (2022), 1-16.
  37. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021), 100336.
  38. Kenny Peng, Arunesh Mathur, and Arvind Narayanan. 2021. Mitigating dataset harms requires stewardship: Lessons from 1000 papers. arXiv preprint arXiv:2108.02922 (2021). · arXiv
  39. Robert N Proctor, Robert Proctor, et al. 1991. Value-free science?: Purity and power in modern knowledge. Harvard University Press.
  40. Inioluwa Deborah Raji and Genevieve Fried. 2021. About face: A survey of facial recognition evaluation. arXiv preprint arXiv:2102.00813 (2021). · arXiv
  41. Urbano Reviglio and Rogers Alunge. 2020. "I am datafied because we are datafied": An Ubuntu perspective on (relational) privacy. Philosophy & Technology 33, 4 (2020), 595-612.
  42. Neil M. Richards. 2013. The Dangers of Surveillance. Harvard Law Review (2013).
  43. Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021. Do datasets have politics? Disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-37.
  44. Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web. 243-246.
  45. Luke Stark. 2019. Facial recognition is the plutonium of AI. XRDS: Crossroads, The ACM Magazine for Students 25, 3 (2019), 50-55.
  46. Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas. 2013. Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nursing & health sciences 15, 3 (2013), 398-405.
  47. Sarah Myers West. 2019. Data capitalism: Redefining the logics of surveillance and privacy. Business & society 58, 1 (2019), 20-41.

Cited by

Sorted by citations

Metadata

Authors
Kalluri, P. R., Agnew, W., Cheng, M., Owens, K., Soldaini, L., & Birhane, A.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2309.15084
arXiv
2309.15084
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review