Making sense of recommendations
Abstract
Computer algorithms are increasingly being used to predict people's preferences and make recommendations. Although people frequently encounter these algorithms because they are cheap to scale, we do not know how they compare to human judgment. Here, we compare computer recommender systems to human recommenders in a domain that affords humans many advantages: predicting which jokes people will find funny. We find that recommender systems outperform humans, whether strangers, friends, or family. Yet people are averse to relying on these recommender systems. This aversion partly stems from the fact that people believe the human recommendation process is easier to understand. It is not enough for recommender systems to be accurate, they must also be understood.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
People are generally averse to using algorithms for tasks usually done by humans, because algorithms are perceived as unable to learn and improve from mistakes, unsuitable for subjective or experiential tasks, unable to adapt to unique or mutable circumstances, and unable to carry responsibility for negative outcomes.
“The literature on algorithm aversion finds that people are generally averse to using algorithms for tasks that are usually done by humans 34,35 , on the grounds that algorithms are perceived as unable to learn and improve from their mistakes 36 , unsuitable for subjective or experiential tasks 37,38 , unable to adapt to unique or mutable circumstances 10,39 and unable to carry responsibility for”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [35]
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317 · cited 2× · see citing works
“231 R CoreTeam (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https://www. R‐project.org/ Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications of the ACM, 40(3), 56–58. https://doi.org/10.1145/245108.245121 Romero, E. J., &…”
BackgroundInfluentialAdomavičius, G., & Tuzhilin, A. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.99 · Cited by 10,347 · see citing works
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135 · see citing works
Evaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798 · see citing works
Empirical Analysis of Predictive Algorithms for Collaborative FilteringPreprint
Breese, J. S., Heckerman, D. E., & Kadie, C. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7363 · Cited by 4,768 · see citing works
mediation : R Package for Causal Mediation Analysis
Tingley, D. H., Yamamoto, T., Hirose, K., Keele, L. J., & Imai, K. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i05 · Cited by 4,216 · see citing works
Resnick, P., & Varian, H. R. (1997) · Communications of the ACM · doi:10.1145/245108.245121 · Cited by 3,772 · see citing works
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · see citing works
Conditions for intuitive expertise: A failure to disagree.
Kahneman, D., & Klein, G. (2009) · American Psychologist · doi:10.1037/a0016755 · Cited by 2,443 · see citing works
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · see citing works
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,187 · see citing works
Combining forecasts: A review and annotated bibliography
Clemen, R. T. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90012-5 · Cited by 2,172 · see citing works
Martin, R. A., Puhlik-Doris, P., Larsen, G., Gray, J., & Weir, K. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(02)00534-2 · Cited by 2,151 · see citing works
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,970 · see citing works
Content-based Recommender Systems: State of the Art and Trends
Lops, P., de Gemmis, M., & Semeraro, G. M. (2010) · Springer eBooks · doi:10.1007/978-0-387-85820-3_3 · Cited by 1,941 · see citing works
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · see citing works
The psychology of humor: an integrative approach
Martin, R. A., & Ford, T. E. (2008) · Choice Reviews Online · doi:10.5860/choice.45-2902 · Cited by 1,700 · see citing works
Clinical versus mechanical prediction: a meta-analysis
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000) · Psychological Assessment · doi:10.1037/1040-3590.12.1.19 · Cited by 1,640 · see citing works
Combining forecasts: A review and annotated bibliography
(1990) · Long Range Planning · doi:10.1016/0024-6301(90)90316-v · Cited by 1,508 · see citing works
Word of mouth and interpersonal communication: A review and directions for future research
Berger, J. (2014) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2014.05.002 · Cited by 1,469 · see citing works
Eigentaste: A Constant Time Collaborative Filtering Algorithm
Goldberg, K., Roeder, T., Gupta, D., et al. (2001) · Information Retrieval · doi:10.1023/a:1011419012209 · Cited by 1,445 · see citing works
The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle
Waytz, A., Heafner, J., & Epley, N. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2014.01.005 · Cited by 1,439 · see citing works
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · see citing works
The base-rate fallacy in probability judgments
Bar‐Hillel, M. (1980) · Acta Psychologica · doi:10.1016/0001-6918(80)90046-3 · Cited by 1,184 · see citing works
GALTON, F. (1907) · Nature · doi:10.1038/075450a0 · Cited by 1,080 · see citing works
McNee, S. M., Riedl, J., & Konstan, J. A. (2006) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1125451.1125659 · Cited by 1,076 · see citing works
The misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032 · see citing works
Bonaccio, S., & Dalal, R. S. (2006) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.07.001 · Cited by 1,011 · see citing works
Advances in Collaborative Filtering
Koren, Y., & Bell, R. (2010) · Springer eBooks · doi:10.1007/978-0-387-85820-3_5 · Cited by 852 · see citing works
Brynjolfsson, E., Hu, Y. (., & Simester, D. (2011) · Management Science · doi:10.1287/mnsc.1110.1371 · Cited by 847 · see citing works
Causes and consequences of mind perception
Waytz, A., Gray, K., Epley, N., & Wegner, D. M. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.006 · Cited by 759 · see citing works
Lessons from the Netflix prize challenge
Bell, R. M., & Koren, Y. (2007) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1345448.1345465 · Cited by 727 · see citing works
Internet Recommendation Systems
Ansari, A., Essegaier, S., & Kohli, R. (2000) · Journal of Marketing Research · doi:10.1509/jmkr.37.3.363.18779 · Cited by 711 · see citing works
The Use of Humor in the Workplace
Romero, E. J., & Cruthirds, K. W. (2006) · Academy of Management Perspectives · doi:10.5465/amp.2006.20591005 · Cited by 541 · see citing works
Comparing Recommendations Made by Online Systems and Friends.
Sinha, R. R., & Swearingen, K. (2001) · Cited by 427 · see citing works
Intuitions about combining opinions: misappreciation of the averaging principle
Larrick, R. P., & Soll, J. B. (2006) · Management Science · doi:10.1287/mnsc.1050.0459 · Cited by 410 · see citing works
Stimulating Online Reviews by Combining Financial Incentives and Social Norms
Burtch, G., Hong, Y., Bapna, R., & Griskevicius, V. (2016) · Management Science · doi:10.1287/mnsc.2016.2715 · Cited by 399 · see citing works
Designing and Evaluating Explanations for Recommender Systems
Tintarev, N., & Masthoff, J. (2010) · Springer eBooks · doi:10.1007/978-0-387-85820-3_15 · Cited by 326 · see citing works
Collaborative prediction and ranking with non-random missing data
Marlin, B. M., & Zemel, R. S. (2009) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/1639714.1639717 · Cited by 319 · see citing works
An Anchoring and Adjustment Model of Spousal Predictions
Davis, H. L., Hoch, S. J., & Ragsdale, E. K. E. (1986) · Journal of Consumer Research · doi:10.1086/209045 · Cited by 292 · see citing works
Perceived consensus and predictive accuracy: The pros and cons of projection.
Hoch, S. J. (1987) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.53.2.221 · Cited by 283 · see citing works
The Benefit of Additional Opinions
Yaniv, I. (2004) · Current Directions in Psychological Science · doi:10.1111/j.0963-7214.2004.00278.x · Cited by 273 · see citing works
Avery, C. N., Resnick, P., & Zeckhauser, R. (1999) · American Economic Review · doi:10.1257/aer.89.3.564 · Cited by 268 · see citing works
The Labor Illusion: How Operational Transparency Increases Perceived Value
Buell, R. W., & Norton, M. I. (2011) · Management Science · doi:10.1287/mnsc.1110.1376 · Cited by 256 · see citing works
Recommendation Systems with Purchase Data
Bodapati, A. V. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.1.77 · Cited by 235 · see citing works
Folkscience: coarse interpretations of a complex reality
Keil, F. C. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00158-x · Cited by 220 · see citing works
The illusion of knowledge: When more information reduces accuracy and increases confidence
Hall, C. C., Ariss, L., & Todorov, A. T. (2007) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2007.01.003 · Cited by 196 · see citing works
Risky business: When humor increases and decreases status.
Bitterly, T. B., Brooks, A. W., & Schweitzer, M. E. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000079 · Cited by 187 · see citing works
Production and appreciation of humor as sexually selected traits
Bressler, E. R., Martin, R. A., & Balshine, S. (2005) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2005.09.001 · Cited by 179 · see citing works
Are We Wise About the Wisdom of Crowds? The Use of Group Judgments in Belief Revision
Mannes, A. E. (2009) · Management Science · doi:10.1287/mnsc.1090.1031 · Cited by 166 · see citing works
Effects of Online Recommendations on Consumers’ Willingness to Pay
Adomavičius, G., Bockstedt, J. C., Curley, S. P., & Zhang, J. (2017) · Information Systems Research · doi:10.1287/isre.2017.0703 · Cited by 148 · see citing works
Technology Usage and Online Sales: An Empirical Study
De, P., Hu, Y. J., & Rahman, M. S. (2010) · Management Science · doi:10.1287/mnsc.1100.1233 · Cited by 122 · see citing works
Humor, Comedy, and Consumer Behavior
Warren, C., Barsky, A. P., & McGraw, A. P. (2018) · Journal of Consumer Research · doi:10.1093/jcr/ucy015 · Cited by 105 · see citing works
The Surprising Power of Neighborly Advice
Gilbert, D. T., Killingsworth, M. A., Eyre, R. N., & Wilson, T. D. (2009) · Science · doi:10.1126/science.1166632 · Cited by 101 · see citing works
Sharing a joke: The effects of a similar sense of humor on affiliation and altruism
Curry, O. S., & Dunbar, R. (2012) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2012.11.003 · Cited by 92 · see citing works
Explanation Fiends and Foes: How Mechanistic Detail Determines Understanding and Preference
Fernbach, P. M., Sloman, S. A., Louis, R. D. S., & Shube, J. N. (2012) · Journal of Consumer Research · doi:10.1086/667782 · Cited by 91 · see citing works
Lerouge, D., & Warlop, L. (2006) · Journal of Consumer Research · doi:10.1086/508523 · Cited by 66 · see citing works
When Does Humorous Marketing Hurt Brands?
Warren, C., & McGraw, A. P. (2016) · Journal of Marketing Behavior · doi:10.1561/107.00000027 · Cited by 58 · see citing works
Krishnan, V., Narayanashetty, P. K., Nathan, M., et al. (2008) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/1454008.1454042 · Cited by 35 · see citing works
Predicting what we will like: Asking a stranger can be as good as asking a friend
Eggleston, C. M., Wilson, T. D., Lee, M., et al. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.03.005 · Cited by 23 · see citing works
Some Hedonic Consequences of Perspective‐Taking in Recommending
Yeomans, M. (2018) · Journal of Consumer Psychology · doi:10.1002/jcpy.1050 · Cited by 20 · see citing works
Self-citationStudying and Modeling the Connection between People's Preferences and Content Sharing
Sharma, A., & Cosley, D. (2015) · ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW) · doi:10.1145/2675133.2675151 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038 · see citing works
Newman, D. T., Fast, N. J., & Harmon, D. J. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.03.008 · Cited by 550 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · see citing works
Trust and reliance on AI — An experimental study on the extent and costs of overreliance on AI
Klingbeil, A., Grützner, C., & Schreck, P. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108352 · Cited by 450 · see citing works
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406 · see citing works
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · see citing works
Understanding, explaining, and utilizing medical artificial intelligence
Cadario, R., Longoni, C., & Morewedge, C. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01146-0 · Cited by 342 · see citing works
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
Man versus machine: Complex estimates and auditor reliance on artificial intelligence
COMMERFORD, B. P., DENNIS, S. A., JOE, J. R., & ULLA, J. W. (2022) · Journal of Accounting Research · doi:10.1111/1475-679x.12407 · Cited by 270 · see citing works
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263 · see citing works
“, 2017), leading them to discount accurate algorithmic recommendations and to rely on inaccurate algorithmic recommendations (Dietvorst et al., 2015; Goodwin & Fildes, 1999; Lim & O'Connor, 1995; Springer et al., 2017; Yeomans et al., 2017).”
BackgroundKeding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168 · see citing works
Wang, C., Li, Y., Fu, W., & Jin, J. (2023) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103325 · Cited by 126 · see citing works
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132 · see citing works
Wu, L., & Jing Wen, T. (2021) · Journal of Advertising Research · doi:10.2501/jar-2021-004 · Cited by 127 · see citing works
Chugunova, M., & Sele, D. (2022) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2022.101897 · Cited by 118 · see citing works
Green, B., & Chen, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479562 · Cited by 69 · see citing works
You’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · see citing works
OK computer: Worker perceptions of algorithmic recruitment
Fumagalli, E., Rezaei, S., & Salomons, A. (2022) · Research Policy · doi:10.1016/j.respol.2021.104420 · Cited by 46 · see citing works
Chiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42 · see citing works
Kaufmann, E. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100028 · Cited by 36 · see citing works
The Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259 · see citing works
Quantifying machine influence over human forecasters
Abeliuk, A., Benjamin, D. M., Morstatter, F., & Galstyan, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-72690-4 · Cited by 18 · see citing works
Behavioral challenges to professional skepticism in auditors’ data analytics journey
Li, X. (2022) · Maandblad voor Accountancy en Bedrijfseconomie · doi:10.5117/mab.96.78525 · Cited by 11 · see citing works
L'acte administratif algorithmique
Stepanov, A. (2026) · doi:10.70675/adc05be3za62ez4b4fz81e7zabf744e60dbf
- Authors
- Yeomans, M., Shah, A., Mullainathan, S., & Kleinberg, J.
- Affiliations
- Harvard University (US)
- University of Chicago (US)
- Cornell University (US)
- Corresponding author
- Yeomans, Michael
- Venue
- Journal of Behavioral Decision Making
- Publisher
- Wiley
- Published
- 2019-02-14
- Volume
- 32
- Issue
- 4
- Pages
- 403-414
- ISSN
- 0894-3257 (ISSN-L)
- 1099-0771
- OpenAlex
- W2913854599
- Cited by
- 459
- References
- 64
- Citations by year
- 2026: 79 · 2025: 83 · 2024: 85 · 2023: 66 · 2022: 46 · 2021: 49 · 2020: 31 · 2019: 13 · 2018: 5 · 2017: 2
- FWCI
- 43.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Behavioral Decision Making (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- recommender systems, human judgment, preference prediction, trust in algorithms, explainability, algorithmic transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
