Is seeing believing?: how recommender system interfaces affect users' opinions
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Recommender systems use people's opinions about items in an information domain to help people choose other items. These systems have succeeded in domains as diverse as movies, news articles, Web pages, and wines. The psychological literature on conformity suggests that in the course of helping people make choices, these systems probably affect users' opinions of the items. If opinions are influenced by recommendations, they might be less valuable for making recommendations for other users. Further, manipulators who seek to make the system generate artificially high or low recommendations might benefit if their efforts influence users to change the opinions they contribute to the recommender. We study two aspects of recommender system interfaces that may affect users' opinions: the rating scale and the display of predictions at the time users rate items. We find that users rate fairly consistently across rating scales. Users can be manipulated, though, tending to rate toward the prediction the system shows, whether the prediction is accurate or not. However, users can detect systems that manipulate predictions. We discuss how designers of recommender systems might react to these findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135
Self-citationGroupLens: an open architecture for collaborative filtering of netnews
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., & Riedl, J. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.192905 · Cited by 5,058
Self-citationEmpirical Analysis of Predictive Algorithms for Collaborative Filtering
Breese, J. S., Heckerman, D. E., & Kadie, C. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7363 · Cited by 4,772
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022
Mining the network value of customers
Domingos, P., & Richardson, M. (2001) · Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/502512.502525 · Cited by 2,830
Social information filtering: algorithms for automating “word of mouth”
Shardanand, U., & Maes, P. (1995) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '95 · doi:10.1145/223904.223931 · Cited by 2,823
Effects of Group Pressure Upon the Modification and Distortion of Judgments
Asch, S. E. (1961) · Documents of Gestalt Psychology · doi:10.1525/9780520313514-017 · Cited by 2,644
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731
Self-citationBesanceney, P. H., Guetzkow, H., & White, W. R. (1952) · The American Catholic Sociological Review · doi:10.2307/3707197 · Cited by 1,440
Eigentaste: A Constant Time Collaborative Filtering Algorithm
Goldberg, K., Roeder, T., Gupta, D., & Perkins, C. (2001) · Information Retrieval · doi:10.1023/a:1011419012209 · Cited by 1,438
Cited by
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 6,627
Self-citationEvaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798
“We know from recent work by Cosley et al. [2003] that user satisfaction is decreased when a significant level of error is introduced into a recommender system.”
ResultSelf-citationIntroduction to Recommender Systems Handbook
Ricci, F., Rokach, L., & Shapira, B. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_1 · Cited by 2,378
Polis, G. A., Anderson, W. B., & Holt, R. D. (1997) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.28.1.289 · Cited by 2,362
Collaborative Filtering Recommender Systems
Schafer, J. B., Frankowski, D., Herlocker, J., & Sen, S. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72079-9_9 · Cited by 2,120
E-Commerce Product Recommendation Agents: Use, Characteristics, and Impact1
Xiao, B., & Benbasat, I. (2007) · MIS Quarterly · doi:10.2307/25148784 · Cited by 1,081
Collaborative Filtering Recommender Systems
Ekstrand, M. D. (2011) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000009 · Cited by 1,037
Recommender Systems: Introduction and Challenges
Ricci, F., Rokach, L., & Shapira, B. (2015) · Recommender Systems Handbook · doi:10.1007/978-1-4899-7637-6_1 · Cited by 846
Research-paper recommender systems: a literature survey
Beel, J., Gipp, B., Langer, S., & Breitinger, C. (2016) · International Journal on Digital Libraries · doi:10.1007/s00799-015-0156-0 · Cited by 823
Explaining the user experience of recommender systems
Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9118-4 · Cited by 799
Recommender systems: from algorithms to user experience
Konstan, J. A., & Riedl, J. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9112-x · Cited by 762
Self-citationShilling recommender systems for fun and profit
Lam, S. K., & Riedl, J. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988726 · Cited by 635
Self-citationA Survey of Explanations in Recommender Systems
Tintarev, N., & Masthoff, J. (2007) · 2007 IEEE 23rd International Conference on Data Engineering Workshop · doi:10.1109/icdew.2007.4401070 · Cited by 566
Reinforcement Learning based Recommender Systems: A Survey
Afsar, M. M., Crump, T., & Far, B. (2023) · ACM Computing Surveys · doi:10.1145/3543846 · Cited by 502
Using Social Psychology to Motivate Contributions to Online Communities
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D., Frankowski, D., Terveen, L., Rashid, A. M., et al. (2005) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00273.x · Cited by 493
Self-citationTime-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols
Campos, P. G., Díez, F., & Cantador, I. (2014) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-012-9136-x · Cited by 437
Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption
Napoli, P. M. (2014) · Communication Theory · doi:10.1111/comt.12039 · Cited by 403
tagging, communities, vocabulary, evolution
Sen, S., Lam, S. K., Rashid, A. M., Cosley, D., Frankowski, D., Osterhouse, J., Harper, F. M., & Riedl, J. (2006) · Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work · doi:10.1145/1180875.1180904 · Cited by 395
Self-citationA historical perspective of explainable Artificial Intelligence
Confalonieri, R., Coba, L., Wagner, B., & Besold, T. R. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1391 · Cited by 383
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
“Therefore, improving their transparency and interpretability has attracted broad interest [34, 35, 45, 50, 64, 65], dating back to early work on recommendation systems [13, 30].”
BackgroundEvaluating recommender systems from the user’s perspective: survey of the state of the art
Pu, P., Chen, L., & Hu, R. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9115-7 · Cited by 377
How should I explain? A comparison of different explanation types for recommender systems
Gedikli, F., Jannach, D., & Ge, M. (2014) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2013.12.007 · Cited by 337
Explaining Recommendations: Design and Evaluation
Tintarev, N., & Masthoff, J. (2015) · Recommender Systems Handbook · doi:10.1007/978-1-4899-7637-6_10 · Cited by 329
Designing and Evaluating Explanations for Recommender Systems
Tintarev, N., & Masthoff, J. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_15 · Cited by 322
Using social psychology to motivate contributions to online communities
Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., & Kraut, R. E. (2004) · Proceedings of the 2004 ACM conference on Computer supported cooperative work · doi:10.1145/1031607.1031642 · Cited by 320
Tagsplanations: explaining recommendations using tags
Vig, J., Sen, S., & Riedl, J. (2009) · Proceedings of the 14th international conference on Intelligent user interfaces · doi:10.1145/1502650.1502661 · Cited by 303
Self-citationEvaluating Recommender Systems: Survey and Framework
Zangerle, E., & Bauer, C. (2023) · ACM Computing Surveys · doi:10.1145/3556536 · Cited by 283
Flanagin, A. J., Metzger, M. J., Pure, R., Markov, A., & Hartsell, E. (2014) · Electronic Commerce Research · doi:10.1007/s10660-014-9139-2 · Cited by 258
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
“Studies have evaluated the influence that technical artifacts such as personalized recommendations, chatbots, and choice architectures have on users’ decision-making [15, 35, 50, 72].”
BackgroundHow behavioural sciences can promote truth, autonomy and democratic discourse online
Lorenz-Spreen, P., Lewandowsky, S., Sunstein, C. R., & Hertwig, R. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0889-7 · Cited by 223
“For instance, feedback loops are created that can artificially reinforce preferences 75–78 , and recommender systems can eliminate context in order to avoid overburdening users.”
BackgroundRecommender Systems: Techniques, Applications, and Challenges
Ricci, F., Rokach, L., & Shapira, B. (2022) · Recommender Systems Handbook · doi:10.1007/978-1-0716-2197-4_1 · Cited by 214
A Reinforcement Learning Framework for Explainable Recommendation
Wang, X., Chen, Y., Yang, J., Wu, L., Wu, Z., & Xie, X. (2018) · 2018 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2018.00074 · Cited by 163
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
My Mobile Music: An Adaptive Personalization System for Digital Audio Players
Chung, T. S., Rust, R. T., & Wedel, M. (2009) · Marketing Science · doi:10.1287/mksc.1080.0371 · Cited by 119
An Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125
Age differences in credibility judgments of online health information
Liao, Q. V., & Fu, W.-T. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2534410 · Cited by 71
Middle-Aged Video Consumers' Beliefs About Algorithmic Recommendations on YouTube
Alvarado, O., Heuer, H., Vanden Abeele, V., Breiter, A., & Verbert, K. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415192 · Cited by 70
Ozok, A. A., Fan, Q., & Norcio, A. F. (2010) · Behaviour & Information Technology · doi:10.1080/01449290903004012 · Cited by 60
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
Jahanbakhsh, F., Katsis, Y., Wang, D., Popa, L., & Muller, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581219 · Cited by 42
Simulation Experiments on (the Absence of) Ratings Bias in Reputation Systems
Thebault-Spieker, J., Kluver, D., Klein, M. A., Halfaker, A., Hecht, B., Terveen, L., & Konstan, J. A. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134736 · Cited by 26
Self-citationEstimating and Penalizing Preference Shift in Recommender Systems
Carroll, M., Hadfield-Menell, D., Russell, S., & Dragan, A. (2021) · Fifteenth ACM Conference on Recommender Systems · doi:10.1145/3460231.3478849 · Cited by 20
A study of dynamic information display and decision-making in abstract trust games
Schaffer, J., O’Donovan, J., Marusich, L., Yu, M., Gonzalez, C., & Höllerer, T. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.002 · Cited by 10
Reviewing the reviewers: A study of author perception on peer reviews in computer science
Kühne, C., Böhm, K., & Yue, J. Z. (2010) · Proceedings of the 6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing · doi:10.4108/icst.collaboratecom.2010.33 · Cited by 8
Metadata
- Authors
- Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., & Riedl, J.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2003-04-05
- Pages
- 585-592
- OpenAlex
- W2116354394
- Cited by
- 467
- References
- 17
- FWCI
- 7.74
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.