The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Thinking critically about and researching algorithms
Abstract
More and more aspects of our everyday lives are being mediated, augmented, produced and regulated by software-enabled technologies. Software is fundamentally composed of algorithms: sets of defined steps structured to process instructions/data to produce an output. This paper synthesises and extends emerging critical thinking about algorithms and considers how best to research them in practice. Four main arguments are developed. First, there is a pressing need to focus critical and empirical attention on algorithms and the work that they do given their increasing importance in shaping social and economic life. Second, algorithms can be conceived in a number of ways – technically, computationally, mathematically, politically, culturally, economically, contextually, materially, philosophically, ethically – but are best understood as being contingent, ontogenetic and performative in nature, and embedded in wider socio-technical assemblages. Third, there are three main challenges that hinder research about algorithms (gaining access to their formulation; they are heterogeneous and embedded in wider systems; their work unfolds contextually and contingently), which require practical and epistemological attention. Fourth, the constitution and work of algorithms can be empirically studied in a number of ways, each of which has strengths and weaknesses that need to be systematically evaluated. Six methodological approaches designed to produce insights into the nature and work of algorithms are critically appraised. It is contended that these methods are best used in combination in order to help overcome epistemological and practical challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
The Data Revolution: Big Data, Open Data, Data Infrastructures & Their Consequences
Kitchin, R. (2014) · doi:10.4135/9781473909472 · Cited by 1,469 · cited 1×
“As are the complex set of decision-making processes and practices, and the wider assemblage of systems of thought, fi nance, politics, legal codes and regulations, materialities and infrastructures, institutions, inter-personal relations, which shape their production (Kitchin, 2014).”
BackgroundSelf-citationThe Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Gillespie, T. (2014) · Media Technologies · doi:10.7551/mitpress/9042.003.0013 · Cited by 1,016
Cited by
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567
The Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911
If...Then: Algorithmic Power and Politics
Bucher, T. (2018) · doi:10.1093/oso/9780190493028.001.0001 · Cited by 1,223
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038
Role of fairness, accountability, and transparency in algorithmic affordance
Shin, D., & Park, Y. J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.019 · Cited by 643
Managing visibility on YouTube through algorithmic gossip
Bishop, S. (2019) · New Media & Society · doi:10.1177/1461444819854731 · Cited by 535
To be or not to be algorithm aware: a question of a new digital divide?
Gran, A.-B., Booth, P., & Bucher, T. (2021) · Information, Communication & Society · doi:10.1080/1369118x.2020.1736124 · Cited by 410
Möller, J., Trilling, D., Helberger, N., & van Es, B. (2018) · Information, Communication & Society · doi:10.1080/1369118x.2018.1444076 · Cited by 387
Swart, J. (2021) · Social Media + Society · doi:10.1177/20563051211008828 · Cited by 381
Algorithmic governance: Developing a research agenda through the power of collective intelligence
Danaher, J., Hogan, M. J., Noone, C., Kennedy, R., Behan, A., De Paor, A., Felzmann, H., Haklay, M., Khoo, S.-M., Morison, J., et al. (2017) · Big Data & Society · doi:10.1177/2053951717726554 · Cited by 369
“This table was created by the lead author based on the discussion in Kitchin (2017).”
BackgroundMethodKatzenbach, C., & Ulbricht, L. (2019) · Internet Policy Review · doi:10.14763/2019.4.1424 · Cited by 359
The Challenges of Algorithm-Based HR Decision-Making for Personal Integrity
Leicht-Deobald, U., Busch, T., Schank, C., Weibel, A., Schafheitle, S., Wildhaber, I., & Kasper, G. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04204-w · Cited by 347
Emerging models of data governance in the age of datafication
Micheli, M., Ponti, M., Craglia, M., & Berti Suman, A. (2020) · Big Data & Society · doi:10.1177/2053951720948087 · Cited by 332
Caplan, R., & Gillespie, T. (2020) · Social Media + Society · doi:10.1177/2056305120936636 · Cited by 283
Transparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260
“Qualitative approaches can help draw detailed and rich accounts of algorithmic model use (Kitchin, 2017).”
BackgroundGrimmelikhuijsen, S. (2023) · Public Administration Review · doi:10.1111/puar.13483 · Cited by 259
Anxiety, panic and self-optimization: Inequalities and the YouTube algorithm
Bishop, S. (2018) · Convergence: The International Journal of Research into New Media Technologies · doi:10.1177/1354856517736978 · Cited by 253
Hannah-Moffat, K. (2019) · Theoretical Criminology · doi:10.1177/1362480618763582 · Cited by 236
Hermann, E. (2022) · New Media & Society · doi:10.1177/14614448211022702 · Cited by 214
Robots, law and the retribution gap
Danaher, J. (2016) · Ethics and Information Technology · doi:10.1007/s10676-016-9403-3 · Cited by 192
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Gil de Zúñiga, H., Goyanes, M., & Durotoye, T. (2024) · Political Communication · doi:10.1080/10584609.2023.2290497 · Cited by 198
Vesnic-Alujevic, L., Nascimento, S., & Pólvora, A. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101961 · Cited by 180
Development and Validation of an Algorithm Literacy Scale for Internet Users
Dogruel, L., Masur, P., & Joeckel, S. (2022) · Communication Methods and Measures · doi:10.1080/19312458.2021.1968361 · Cited by 177
Beyond user experience: What constitutes algorithmic experiences?
Shin, D., Zhong, B., & Biocca, F. A. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.102061 · Cited by 174
Meijer, A., Lorenz, L., & Wessels, M. (2021) · Public Administration Review · doi:10.1111/puar.13391 · Cited by 174
Ziolkowski, R., Miscione, G., & Schwabe, G. (2020) · Journal of Management Information Systems · doi:10.1080/07421222.2020.1759974 · Cited by 151
Cotter, K. (2023) · Information, Communication & Society · doi:10.1080/1369118x.2021.1994624 · Cited by 149
Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149
Generalised scepticism: how people navigate news on social media
Fletcher, R., & Nielsen, R. K. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1450887 · Cited by 146
Bonini, T., & Gandini, A. (2019) · Social Media + Society · doi:10.1177/2056305119880006 · Cited by 146
International migration management in the age of artificial intelligence
Beduschi, A. (2021) · Migration Studies · doi:10.1093/migration/mnaa003 · Cited by 141
Ranerup, A., & Henriksen, H. Z. (2022) · Social Science Computer Review · doi:10.1177/0894439320980434 · Cited by 139
Addressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138
Liable, but not in control? Ensuring meaningful human agency in automated decision-making systems
Wagner, B. (2019) · Policy & Internet · doi:10.1002/poi3.198 · Cited by 136
“This follows a trend related to the algorithms and automation literature which both questions and emphasizes the importance of human agency within otherwise vastly automated and mostly Internet-connected technical systems (Gillespie, 2014; Kitchin, 2017; Pasquale, 2015).”
BackgroundAlgorithms and Taste-Making: Exposing the Netflix Recommender System’s Operational Logics
Pajkovic, N. (2022) · Convergence: The International Journal of Research into New Media Technologies · doi:10.1177/13548565211014464 · Cited by 125
The Superhost. Biopolitics, home and community in the Airbnb dream-world of global hospitality
Roelofsen, M., & Minca, C. (2018) · Geoforum · doi:10.1016/j.geoforum.2018.02.021 · Cited by 123
Algorithmic Literacy and the Role for Libraries
Ridley, M., & Pawlick-Potts, D. (2021) · Information Technology and Libraries · doi:10.6017/ital.v40i2.12963 · Cited by 111
Kaun, A. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2021.1924827 · Cited by 97
Shao, C., Nah, S., Makady, H., & McNealy, J. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2331858 · Cited by 94
Bountouridis, D., Harambam, J., Makhortykh, M., Marrero, M., Tintarev, N., & Hauff, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287583 · Cited by 92
Dencik, L., Hintz, A., & Carey, Z. (2018) · New Media & Society · doi:10.1177/1461444817697722 · Cited by 90
Automated Trouble: The Role of Algorithmic Selection in Harms on Social Media Platforms
Saurwein, F., & Spencer-Smith, C. (2021) · Media and Communication · doi:10.17645/mac.v9i4.4062 · Cited by 88
ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia
Halfaker, A., & Geiger, R. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415219 · Cited by 82
Koenig, A. (2020) · Computers and Composition · doi:10.1016/j.compcom.2020.102611 · Cited by 69
Cotter, K., Medeiros, M., Pak, C., & Thorson, K. (2021) · Big Data & Society · doi:10.1177/2053951721996046 · Cited by 63
Voorveld, H. A. M., Meppelink, C. S., & Boerman, S. C. (2024) · International Journal of Advertising · doi:10.1080/02650487.2023.2264045 · Cited by 62
Repackaging authority: artificial intelligence, automated governance and education trade shows
Gulson, K. N., & Witzenberger, K. (2022) · Journal of Education Policy · doi:10.1080/02680939.2020.1785552 · Cited by 60
The Internet Archive and the socio-technical construction of historical facts
Ben-David, A., & Amram, A. (2018) · Internet Histories · doi:10.1080/24701475.2018.1455412 · Cited by 60
Shin, D. (. (2019) · Javnost - The Public · doi:10.1080/13183222.2019.1589249 · Cited by 53
Do Algorithms Shape Character? Considering Algorithmic Ethical Subjectivation
Magalhães, J. C. (2018) · Social Media + Society · doi:10.1177/2056305118768301 · Cited by 52
Hagendorff, T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09573-8 · Cited by 47
The return of the social: Algorithmic identity in an age of symbolic demise
Kotliar, D. M. (2020) · New Media & Society · doi:10.1177/1461444820912535 · Cited by 45
Regulating AI and Robotics: Ethical and Legal Challenges
Ebers, M. (2020) · Algorithms and Law · doi:10.1017/9781108347846.003 · Cited by 44
Putting AI Ethics into Practice: The Hourglass Model of Organizational AI Governance
Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022) · arXiv · doi:10.48550/arxiv.2206.00335 · Cited by 38
What rules? Framing the governance of artificial agency
Gahnberg, C. (2021) · Policy and Society · doi:10.1080/14494035.2021.1929729 · Cited by 35
Holm, S., Stanton, C., & Bartlett, B. (2021) · Health Care Analysis · doi:10.1007/s10728-021-00430-4 · Cited by 32
Lundahl, O. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1864006 · Cited by 29
Artificial Intelligence and Legal Tech: Challenges to the Rule of Law
Buchholtz, G. (2020) · Regulating Artificial Intelligence · doi:10.1007/978-3-030-32361-5_8 · Cited by 27
Sarra, C. (2020) · Global Jurist · doi:10.1515/gj-2020-0003 · Cited by 24
Algorithms and the narration of past selves
Jacobsen, B. N. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1834603 · Cited by 24
Evolving Coagency between Artists and AI in the Spatial Cocreative Process of Artmaking
Nordström, P., Lundman, R., & Hautala, J. (2023) · Annals of the American Association of Geographers · doi:10.1080/24694452.2023.2210647 · Cited by 17
Marjanovic, O., Cecez-Kecmanovic, D., & Vidgen, R. (2018) · IFIP Advances in Information and Communication Technology · doi:10.1007/978-3-030-04091-8_4 · Cited by 15
Digitising “The Big Lie”: Algorithmic Curation as an Inhibitor of Media Exposure Diversity Online
Reid, J. (2024) · Communicatio · doi:10.1080/02500167.2024.2424841 · Cited by 10
23 Die Plattformisierung des Erinnerns
Smit, R. (2022) · Handbuch kommunikationswissenschaftliche Erinnerungsforschung · doi:10.1515/9783110629743-023 · Cited by 9
Towards a decolonial AI sovereignty model for indigenous educational settings
Mbah, M. F., & Jarvis, L. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2626631 · Cited by 7
Creative ML Assemblages: The Interactive Politics of People, Processes, and Products
Shelby, R., Srinivasan, R., Burgdorf, K., Lena, J. C., & Rostamzadeh, N. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637315 · Cited by 6
L'acte administratif algorithmique
Stepanov, A. (2026) · doi:10.70675/adc05be3za62ez4b4fz81e7zabf744e60dbf
Metadata
- Authors
- Kitchin, R.
- Venue
- Information, Communication & Society
- Publisher
- Informa UK Limited
- Published
- 2016-02-25
- Volume
- 20
- Issue
- 1
- Pages
- 14-29
- ISSN
- 1369-118X (ISSN-L)
- 1468-4462
- OpenAlex
- W1515490832
- Cited by
- 1,186
- References
- 4
- FWCI
- 168.72
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.