Peer review and competition in the Art Exhibition Game
Open accessAbstract
Significance: Competition is an essential mechanism in increasing the effort and performance of human groups in real life. However, competition has side effects: it can be detrimental to creativity and reduce cooperation. We conducted an experiment called the Art Exhibition Game to investigate the effect of competitive incentives in environments where the quality of creative products and the amount of innovation allowed are decided through peer review. Our approach is general and can provide insights in domains such as clinical evaluations, scientific admissibility, and science funding. Our results show that competition leads to more innovation but also to more unfair reviews and to a lower level of agreement between reviewers. Moreover, competition does not improve the average quality of published works.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Dishonest behavior is a prominent problem in reviews of papers that they were unable to study in the context of reviewing reviews.
“There is one prominent problem which exists in reviews of papers which we are unable to study in the context of reviewing reviews—dishonest behavior.”
From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [41]
The paper’s section on review fairness reports that competition led participants in the Art Exhibition Game to give unfairly low scores to direct competitors. That supports dishonest or self-interested reviewing as a concern, but the Discussion says the experiment used abstracted art exhibitions and that its results may be transferable to scholarly peer review; it does not establish the behavior’s prevalence in reviews of papers. The citing authors’ inability to study it is their own framing, not a finding attributed to this source. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,219 · cited 1× · see citing works
“It is important to note that, unlike in previous studies of diversity in artificial cultural markets (25), the participants in the Art Exhibition Game are not only passive evaluators but also produce similar kinds of artwork themselves, meaning that they constantly adapt their creative output to the feedback they rece…”
BackgroundMisconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232 · cited 1× · see citing works
“In times where science is increasingly witnessing peer review rings, scientific fraud, and plagiarism (40, 45, 48, 49), the results of our study suggest a redesign the scientific incentive system such that sustainable forms of competition are promoted.”
ResultOn the fragility of skilled performance: What governs choking under pressure?
Beilock, S. L., & Carr, T. H. (2001) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.130.4.701 · Cited by 1,201 · cited 1× · see citing works
“Similarly, competitive pressure can reduce the performance of professional athletes, causing them to “choke under pressure” (8, 9).”
BackgroundIncentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis
Boudreau, K. J., Lacetera, N., & Lakhani, K. R. (2011) · Management Science · doi:10.1287/mnsc.1110.1322 · Cited by 830 · cited 3× · see citing works
“Proc Natl Acad Sci USA 109(42):17028–17033.”
BackgroundResultInfluentialRescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · see citing works
Dynamical evolutionary psychology: Individual decision rules and emergent social norms.
Kenrick, D. T., Li, N. P., & Butner, J. (2003) · Psychological Review · doi:10.1037/0033-295x.110.1.3 · Cited by 358 · see citing works
Mason, W. A., Conrey, F. R., & Smith, E. R. (2007) · Personality and Social Psychology Review · doi:10.1177/1088868307301032 · Cited by 335 · see citing works
Fiske, D. W., & Fogg, L. F. (1990) · American Psychologist · doi:10.1037/0003-066x.45.5.591 · Cited by 142 · see citing works
The luck of the referee draw: the effect of exchanging reviews
BORNMANN, L., & DANIEL, H. (2009) · Learned Publishing · doi:10.1087/2009207 · Cited by 51 · see citing works
Flagged first, then by citations
Research works printed but not linked: 37
- Bull C, Schotter A, Weigelt K (1987) Tournaments and piece rates: An experimental study. J Polit Econ 95(1):1-33.
- Bonner SE, Sprinkle G (2002) The effects of monetary incentives on effort and task performance: Theories, evidence, and a framework for research. Account Organ Soc 27(4):303-345.
- Byron K, Khazanchi S (2012) Rewards and creative performance: A meta-analytic test of theoretically derived hypotheses. Psychol Bull 138(4):809-830.
- Gneezy U, Rustichini A (2000) A fine is a price. J Legal Stud 29(1):1-17.
- Gagné M, Deci E (2005) Self-determination theory and work motivation. J Organ Behav 26(4):331-362.
- Baumeister R, Showers C (1986) A review of paradoxical performance effects: Choking under pressure in sports and mental tests. Eur J Soc Psychol 16(4):361-383.
- Harbring C, Irlenbusch B (2008) How many winners are good to have?: On tournaments with sabotage. J Econ Behav Organ 65(3):682-702.
- Gürtler O, Münster J (2010) Sabotage in dynamic tournaments. J Math Econ 46(2): 179-190.
- Helbing D (2007) Managing Complexity: Insights, Concepts, Applications (Springer, New York).
- Scott W (1974) Interreferee agreement on some characteristics of manuscripts sub- mitted to the journal of personality and social psychology. Am Psychol 29(9):698-702.
- Rothwell PM, Martyn CN (2000) Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain 123(Pt 9):1964-1969.
- Mahoney M (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognit Ther Res 1(2):161-175.
- Mutz R, Bornmann L, Daniel HD (2015) Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data ap- proach using ex-ante and ex-post peer evaluation data from the austrian science fund. J Assoc Inf Sci Technol 66(11):2321-2339.
- Thurner S, Hanel R (2011) Peer-review in a world with rational scientists: Toward selection of the average. Eur Phys J B 84(4):707-711.
- Squazzoni F, Gandelli C (2012) Saint matthew strikes again: An agent-based model of peer review and the scientific community structure. J Informetrics 6(2):265-275.
- Thorngate W, Chowdhury W (2014) Advances in Social Simulation (Springer, New York), pp 177-188.
- Amabile TM, Hennessey BA, Grossman BS (1986) Social influences on creativity: the effects of contracted-for reward. J Pers Soc Psychol 50(1):14-23.
- Bornmann L, Mutz R, Marx W, Schier H, Daniel HD (2011) A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high profile journal select manuscripts that are highly cited after publication? J R Stat Soc Ser A Stat Soc 174(4):857-879.
- Chernoff H (1973) The use of faces to represent points in K-dimensional space graphically. J Am Stat Assoc 68(342):361-368.
- Sprouse J (2011) A validation of Amazon Mechanical Turk for the collection of accept- ability judgments in linguistic theory. Behav Res Methods 43(1):155-167.
- Sawyer R (2011) Explaining Creativity: The Science of Human Innovation (Oxford Univ Press, Oxford, UK).
- Fang F, Casadevall A (2015) Competitive science: Is competition ruining science? Infect Immun 83(4):1229-1233.
- Andreasen NC, Ramchandran K (2012) Creativity in art and science: Are there two cultures? Dialogues Clin Neurosci 14(1):49-54.
- Amabile T (1983) The Social Psychology of Creativity (Springer-Verlag, Berlin).
- Simonton D (1999) Creativity as blind variation and selective retention: Is the creative process darwinian? Psychol Inq 10(4):309-328.
- Simonton DK (2009) Varieties of (scientific) creativity a hierarchical model of domain- specific disposition, development, and achievement. Perspect Psychol Sci 4(5):441-452.
- Simon H (2001) Creativity in the arts and the sciences. Kenyon Review 23(2):203-220.
- Latane B, Bourgeois M (2001) Handbook of social psychology. Group Processes, eds Tindale R, Hogg M (Blackwell, Malden, MA), Vol 4, pp 235-258.
- Lorenz J, Rauhut H, Schweitzer F, Helbing D (2011) How social influence can undermine the wisdom of crowd effect. Proc Natl Acad Sci USA 108(22):9020-9025.
- Petty R, Fleming MA, Fabrigar L (1999) The review process at pspb: Correlates of in- terreviewer agreement and manuscript acceptance. Pers Soc Psychol Bull 25(2):188-203.
- Thorngate W, Dawes R, Foddy M (2011) Judging Merit (Psychology Press, New York).
- Ferguson C, Marcus A, Oransky I (2014) Publishing: The peer-review scam. Nature 515(7528):480-482.
- Anderson MS, Ronning EA, De Vries R, Martinson BC (2007) The perverse effects of competition on scientists' work and relationships. Sci Eng Ethics 13(4):437-461.
- Siler K, Lee K, Bero L (2015) Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci USA 112(2):360-365.
- Wuchty S, Jones BF, Uzzi B (2007) The increasing dominance of teams in production of knowledge. Science 316(5827):1036-1039.
- Butler D (2010) Journals step up plagiarism policing. Nature 466(7303):167.
- Azoulay P, Graff Zivin J, Manso G (2011) Incentives and creativity: Evidence from the academic life sciences. Rand J Econ 42(3):527-554.
Cited by
Sorted by citations
Challenges, experiments, and computational solutions in peer review
Shah, N. B. (2022) · Communications of the ACM · doi:10.1145/3528086 · Cited by 66 · see citing works
Peer reviews of peer reviews: A randomized controlled trial and other experiments
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B. (2025) · PLOS ONE · doi:10.1371/journal.pone.0320444 · Cited by 18 · see citing works
“One form of dishonest behavior is that of “lone wolf” dishonesty in which reviewers, who are also authors of some submitted papers, deliberately manipulate the reviews they provide to increase the chances of their own papers being accepted [BGH16, XZSS19, DJKS22].”
- Authors
- Balietti, S., Goldstone, R. L., & Helbing, D.
- Affiliations
- Northeastern University (US)
- Indiana University Bloomington (US)
- ETH Zurich (CH)
- Corresponding author
- Balietti, Stefano
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2016-07-11
- Volume
- 113
- Issue
- 30
- Pages
- 8414-8419
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4968744
- OpenAlex
- W2473242869
- Cited by
- 71
- References
- 9
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 5 · 2023: 4 · 2022: 5 · 2021: 15 · 2020: 9 · 2019: 5 · 2018: 10 · 2017: 4 · 2016: 2
- FWCI
- 17.08
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- MADOC (University of Mannheim) (acceptedVersion)
- Funders
- U.S. Army Research Office (W911NF–14–1–0478)
- EC | European Research Council (324247)
- Topics
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- peer review, competition, innovation, product variety, reviewer agreement, rejection rate, creativity, scholarly publishing
- MeSH terms
- Competitive Behavior, Creativity, Games, Experimental, Humans, Interpersonal Relations, Peer Review, Photic Stimulation, Surveys and Questionnaires
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
