Black box measures? How to study people’s algorithm skills
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Considerable scholarship has established that algorithms are an increasingly important part of what information people encounter in everyday life. Much less work has focused on studying users’ experiences with, understandings of, and attitudes about how algorithms may influence what they see and do. The dearth of research on this topic globally with diverse populations may be in part due to the difficulty of studying a subject about which there is no known ground truth given that details about algorithms are proprietary and rarely made public. This paper explicitly takes on the methodological challenges of studying people’s algorithm skills to shed light on the special considerations required when studying a topic about which even the researchers possess limited know-how. The paper advocates for more such scholarship to accompany existing system-level analyses of algorithms’ social implications and offers a blueprint for how to do so.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · cited 1×
“Understanding algorithms better can have similar consequences, but not just in the realm of what Web pages rise to the top of a search engine ’ s results page, which we know from research is what the vast majority of people rely on for their information source (e.g., Pan et al., 2007).”
BackgroundStriphas, T. (2015) · European Journal of Cultural Studies · doi:10.1177/1367549415577392 · Cited by 647 · cited 1×
“Much has been written about the many concerns raised by such opaque systems from Web sites to apps (e.g., Dia-kopoulos, 2015; Just & Latzer, 2016; Pasquale, 2015; Striphas, 2015) including their potential for bias and discrimination (Eubanks, 2018; Noble, 2018; O ’ Neil, 2016).”
BackgroundFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375
DeVito, M. A., Gergle, D., & Birnholtz, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025659 · Cited by 311
Succinct survey measures of web-use skills
Hargittai, E., & Hsieh, Y. P. (2012) · Social Science Computer Review · doi:10.1177/0894439310397146 · Cited by 288 · cited 1×
“The goal of this instrument was to gather some general Internet skill measures using previously established instruments (Hargittai & Hsieh, 2012; Park, 2013) and collect basic demographics about respondents (year of birth, gender, education, and employment status).”
MethodSelf-citationWeb credibility assessment: Conceptualization, operationalization, variability, and models
Choi, W., & Stvilia, B. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23543 · Cited by 168 · cited 1×
“First, existing literature (see above) tends to focus on social media and search engine use (see Choi & Stvilia, 2015 for a review of the related credibility-assessment literature), which we wanted to extend to other domains.”
BackgroundInternet Skills and Why They Matter
Hargittai, E., & Micheli, M. (2019) · Society and the Internet · doi:10.1093/oso/9780198843498.003.0007 · Cited by 99 · cited 2×
“Algorithm skills can be seen as one type of Internet skill and refers here to people ’ s awareness and understanding of the systems that operate behind-the-scenes to bring content to users (Hargittai & Micheli, 2019).”
BackgroundSelf-citation
Cited by
Swart, J. (2021) · Social Media + Society · doi:10.1177/20563051211008828 · Cited by 381
“The more bluntly users are categorized and classified, the more “expectancy violations” (Hargittai et al., 2020) occur, and thus the more visible algorithms become.”
BackgroundMethodInfluentialDevelopment and Validation of an Algorithm Literacy Scale for Internet Users
Dogruel, L., Masur, P., & Joeckel, S. (2022) · Communication Methods and Measures · doi:10.1080/19312458.2021.1968361 · Cited by 177
“Adopting a qualitative approach, yet focusing on a broad measure toward algorithm literacy, Hargittai et al. (2020) relied on an interview protocol investigating internet users’ awareness of curation and personalization in different internet services (e.g., search engines, news sites, social media,…”
BackgroundResultInfluentialAdaptive Folk Theorization as a Path to Algorithmic Literacy on Changing Platforms
DeVito, M. A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476080 · Cited by 151
”Because AI is 100% right and safe”: User Attitudes and Sources of AI Authority in India
Kapania, S., Siy, O., Clapper, G., SP, A. M., & Sambasivan, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517533 · Cited by 90
“Algorithmic skills involve the ability to identify that a dynamic system is in place (awareness), and reflecting on how those systems work (understanding) [45].”
BackgroundInfluentialNot All AI are Equal: Exploring the Accessibility of AI-Mediated Communication Technology
Goldenthal, E., Park, J., Liu, S. X., Mieczkowski, H., & Hancock, J. T. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106975 · Cited by 79
Metadata
- Authors
- Hargittai, E., Gruber, J., Djukaric, T., Fuchs, J., & Brombach, L.
- Venue
- Information, Communication & Society
- Publisher
- Informa UK Limited
- Published
- 2020-03-23
- Volume
- 23
- Issue
- 5
- Pages
- 764-775
- ISSN
- 1369-118X (ISSN-L)
- 1468-4462
- OpenAlex
- W3014032479
- Cited by
- 144
- References
- 7
- FWCI
- 19.28
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Black box measures? How to study people’s algorithm skills. https://app.cochat.ai/papers/10.1080/1369118x.2020.1713846
- The paper
- Hargittai, E., Gruber, J., Djukaric, T., Fuchs, J., & Brombach, L. (2020). Black box measures? How to study people’s algorithm skills. Information, Communication & Society. https://doi.org/10.1080/1369118x.2020.1713846
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.