User attitudes towards algorithmic opacity and transparency in online reviewing platforms
Abstract
Algorithms exert great power in curating online information, yet are often opaque in their operation, and even existence. Since opaque algorithms sometimes make biased or deceptive decisions, many have called for increased transparency. However, little is known about how users perceive and interact with potentially biased and deceptive opaque algorithms. What factors are associated with these perceptions, and how does adding transparency into algorithmic systems change user attitudes? To address these questions, we conducted two studies: 1) an analysis of 242 users' online discussions about the Yelp review filtering algorithm and 2) an interview study with 15 Yelp users disclosing the algorithm's existence via a tool. We found that users question or defend this algorithm and its opacity depending on their engagement with and personal gain from the algorithm. We also found adding transparency into the algorithm changed users' attitudes towards the algorithm: users reported their intention to either write for the algorithm in future reviews or leave the platform.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Yelp presents reviews as 'recommended' to the author if they are logged in, but may filter out this review for other users.
“This would show that Yelp presents reviews as”
From Expanding on the Frames: Making a Case for Algorithmic Literacy (2023) · cites this paper as Eslami et al., 2019
Cites
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2× · see citing works
“As Burrell discusses, this opacity stems from 1) corporate secrecy geared to prevent malicious users from gaming the system, 2) the limited technical literacy of regular users of these systems, and 3) the complexity of understanding an algorithm in action, even by its own developers [5].”
BackgroundAnanny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · see citing works
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
Algorithmic Transparency in the News Media
Diakopoulos, N., & Koliska, M. (2017) · Digital Journalism · doi:10.1080/21670811.2016.1208053 · Cited by 515 · see citing works
Explanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · see citing works
Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed
Rader, E., & Gray, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702174 · Cited by 386 · cited 3× · see citing works
“For example, recent work studied users’ of awareness of opaque social media feed curation algorithms [14] and users’ folk theories on how algorithms work [8, 12, 16, 36].”
BackgroundFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375 · see citing works
Self-citationCommunicating Algorithmic Process in Online Behavioral Advertising
Eslami, M., Krishna Kumaran, S. R., Sandvig, C., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174006 · Cited by 171 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · see citing works
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211 · see citing works
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · see citing works
Self-citationBuilding and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · see citing works
Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119 · see citing works
DeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98 · see citing works
Self-citationTo Live in Their Utopia: Why Algorithmic Systems Create Absurd Outcomes
Alkhatib, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445740 · Cited by 81 · see citing works
Vaccaro, K., Sandvig, C., & Karahalios, K. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415238 · Cited by 102 · see citing works
Self-citationOne AI Does Not Fit All: A Cluster Analysis of the Laypeople’s Perception of AI Roles
Kim, T., Molina, M. D., Rheu, M. (., Zhan, E. S., & Peng, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581340 · Cited by 68 · see citing works
Schoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67 · see citing works
SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice
Singhal, M., Ling, C., Paudel, P., Thota, P., Kumarswamy, N., Stringhini, G., & Nilizadeh, S. (2023) · 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp57164.2023.00056 · Cited by 57 · see citing works
Haque, M. R., & Rubya, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555146 · Cited by 48 · see citing works
Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42 · see citing works
Misfitting With AI: How Blind People Verify and Contest AI Errors
Alharbi, R., Lor, P., Herskovitz, J., Schoenebeck, S., & Brewer, R. N. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675659 · Cited by 40 · see citing works
Tensions in transparent urban AI: designing a smart electric vehicle charge point
Alfrink, K., Keller, I., Doorn, N., & Kortuem, G. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01436-9 · Cited by 18 · see citing works
“A level of obfuscation is necessary to prevent bad actors from gaming the system, whereas a lack of transparency reduces people’s sense of control and makes them unsure about how their behavior might impact outcomes (Alvarado and Waern 2018; Jhaver et al. 2018; Eslami et al. 2019).”
BackgroundExpanding on the Frames: Making a Case for Algorithmic Literacy
Loyola Marymount University, & Archambault, S. (2023) · Communications in Information Literacy · doi:10.15760/comminfolit.2023.17.2.11 · Cited by 19 · see citing works
A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making
Schoeffer, J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503811 · Cited by 9 · see citing works
- Authors
- Eslami, M., Vaccaro, K., Lee, M. K., Elazari Bar On, A., Gilbert, E., & Karahalios, K.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Carnegie Mellon University (US)
- University of California, Berkeley, Berkeley, CA, USA
- University of Michigan, Ann Arbor, MI, USA
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-14
- OpenAlex
- W2940680165
- Cited by
- 184
- References
- 8
- Citations by year
- 2026: 20 · 2025: 30 · 2024: 33 · 2023: 34 · 2022: 29 · 2021: 22 · 2020: 13 · 2019: 3
- FWCI
- 17.48
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Google PhD Fellowship
- NSF (CHS-1564041)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- algorithmic opacity, algorithmic transparency, online review platforms, Yelp, user attitudes, algorithmic bias, user engagement, user-generated content
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
