CoChat
Sign Up
Cited by 8Mobile Crowdsensing and CrowdsourcingOpen Source Software InnovationsEducational Games and Gamification

Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests

Open access
Tsvetkova, Milena, Müller, Sebastian, Vuculescu, Oana, Ham, Haylee, Sergeev, Rinat A.·2022·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3555649
Open this paper in CoChat — free accountOpen PDF

Abstract

Rankings and leaderboards are often used in crowdsourcing contests and online communities to motivate individual contributions but feedback based on social comparison can also have negative effects. Here, we study the unequal effects of such feedback on individual effort and performance for individuals of different ability. We hypothesize that the effects of social comparison differ for top performers and bottom performers in a way that the inequality between the two increases. We use a quasi-experimental design to test our predictions with data from Topcoder, a large online crowdsourcing platform that publishes computer programming contests. We find that in contests where the submitted code is evaluated against others' submissions, rather than using an absolute scale, top performers increase their effort while bottom performers decrease it. As a result, relative scoring leads to better outcomes for those at the top but lower engagement for bottom performers. Our findings expose an important but overlooked drawback from using gamified competitions, rankings, and relative evaluations, with potential implications for crowdsourcing markets, online learning environments, online communities, and organizations in general.

AI snapshot

Written from the paper’s full text.

TL;DR

In a quasi-experiment on Topcoder, relative (social-comparison) scoring increased effort and performance for high-rated coders but decreased effort for low-rated coders, widening disparities. The paper supports hypotheses on effort and performance dispersion but finds no significant evidence that relative scoring increases predictability of performance.

Key findings

  • H1 supported: In relatively scored vs. absolutely scored challenges, the bottom 15% by rating wrote/edited 46–245 fewer lines, while the top 15% wrote/edited 300–383 more; the regression difference was significant, ∆χ²(2df)=7.633, p=0.022. Bottom 45% made 1.6–2.5 fewer submissions, ∆χ²(2df)=7.537, p=0.023; duration difference was not significant, ∆χ²(2df)=4.682, p=0.096.
  • H2 supported: Code complexity disparity was greater under relative scoring, ∆χ²(3df)=11.532, p=0.009. The effect was mainly driven by coders above the 50th percentile, who submitted about 30–74 additional independent logical paths; lowest-rated coders did not develop solutions as much.
  • H3 not supported: Predictability of performance from prior performance did not significantly differ, ∆χ²(2df)=3.974, p=0.137. First-submission timing, length, and complexity did not differ significantly, suggesting final outcomes were not driven by a priori expectations or dispositions.

Methods

  • Data: Topcoder marathon matches, covering 189,659 successful code submissions by 17,000 individuals in 367 challenges from 15 December 2005 to 1 April 2018; scoring type was manually classified by coders as absolute vs. relative.
  • Design: Coarsened exact matching on challenge size, duration, difficulty, reward structure, and task yielded 52 challenges in 16 strata. A crossover quasi-experiment restricted to coders with both scoring types in the same stratum and a rating left 1,239 observations of 319 coders over 42 challenges in 12 strata, with 12,914 successful submissions.
  • Analysis: Mixed-effects models with random intercepts for strata, challenges, and individuals; gamma regressions for code lines, submissions, and cyclomatic complexity; beta regressions for time in challenge and proximity to rating; stratum weights and controls for previous challenges; estimand was the local average treatment effect for the treated.

Limitations

  • Stated: Matching and the quasi-experimental crossover design discard most data and narrow the claims; findings may not transfer to contests with monetary rewards; matching cannot entirely preclude confounding, e.g., relative scoring may be applied to more open-ended, less structured challenges; no survey or random assignment was available.
  • Stated: Cyclomatic complexity is limited as a performance measure; it correlates strongly with lines of code (Pearson 0.878), so results are interpreted narrowly as elaboration/development, not maintainability, optimization, or legibility.
  • Stated/evident: The matched sample does not represent the Topcoder population; results concern experienced coders and do not extend to first-time contestants; matched challenges had no monetary or nonmonetary prizes (Table 4), and subjects were more experienced and higher rated than average contestants; Topcoder is not representative, replication is needed, and upward vs. downward comparison or smaller collaboration-centered groups may differ.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3555649

Cites

81 of 81 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

16 of 77 cited sources checked · 2 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Empirical research on contests and tournaments contradicts Festinger's expectation that underachievers increase effort.

    “However, empirical research on contests and tournaments contradicts Festinger's expectation that underachievers increase effort.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Cason et al., 2010 source

      The Results (section 3.2; page unavailable) show that lower-ability subjects were less likely to enter winner-take-all contests when facing strong opponents. The same section reports that individual performance did not depend on competitors’ abilities. These findings do not show underachievers increasing effort, but the paper tests contest entry and performance, not Festinger’s specific expectation about underachievers’ effort. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Dechenaux et al., 2015 source

      The review does not present a general contradiction of the expectation. In §6.2, p. 30, it reports that participants who ranked worse than expected increased their performance and that contestants who lagged behind tended to increase effort. It also describes mixed findings on feedback, including quitting in some tournaments. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites van Dijk, Sonnemans, & van Winden, 2001 source
  • Premisecitation

    Compared to piece-rate schemes, contests, all-pay auctions, and tournaments result in higher variance in individual effort.

    “Compared to piece-rate schemes, contests, all-pay auctions, and tournaments result in higher variance in individual effort (Cason et al., 2010;Dechenaux et al., 2015;van Dijk, Sonnemans, & van Winden, 2001).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Cason et al., 2010 source

      The Introduction (page unavailable) says winner-take-all incentives may produce greater variance in players’ effort than piece-rate wages. Section 2.2 (page unavailable) identifies the paper’s winner-take-all contest as an all-pay auction. The citing sentence overstates a qualified possibility as a general result for contests and tournaments, including designs beyond winner-take-all. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Dechenaux et al., 2015 source

      Section 2.3 (p. 10) describes high variance in individual effort in rank-order tournaments. But it also reports that effort variance was comparable between tournaments and piece-rate schemes when their payoff functions were made similar. The cited paper therefore does not support the unqualified claim that tournaments—and contests and all-pay auctions generally—produce higher effort variance than piece rates. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites van Dijk, Sonnemans, & van Winden, 2001 source

      The Hypothesis 4 results (p. 23) report significantly greater variability in tournaments than in the piece-rate condition. The cited paper does not establish the same comparison for contests generally or all-pay auctions, which the citing sentence also attributes to its citations. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Interpretationcitation

    Those at the top of the chart may experience the opposite effect, whereby a provisional reward for a serendipitous move could increase their confidence and motivation to persevere.

    “Conversely, those at the top of the chart may experience the opposite effect, whereby a provisional reward for a serendipitous move could increase their confidence and motivation to persevere (Restivo & van de Rijt, 2012).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Restivo & van de Rijt, 2012 source

      The Methods describe randomly awarding barnstars to a sample of the top 1% of Wikipedia editors, and Results and Discussion report greater sustained productivity and less discontinuation among recipients (page numbers not provided). This supports the possibility that a reward helps high-performing contributors persevere. The paper does not measure confidence or study rewards for a serendipitous move. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Findingown data
    § Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests: Evidence from a Quasi-Experiment on Topcoder

    Relative feedback increases disparities in effort and performance in crowdsourcing contests.

    “Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests: Evidence from a Quasi-Experiment on Topcoder”
    Quote in the paperSubstantial claim
  • Findingown data

    The difference in number of submissions between relative and absolute scoring by rating stratum is statistically significant (∆χ2(2df)=7.537, p=0.023).

    “The difference for number of submissions is statistically significant (∆χ 2 (2df ) = 7.537, p = 0.023 between Model 2.1 and 2.2 in Table 1)”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 69
  • May combine two claims. 48
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Within-group competition for scarce resources decreases contributions to public goods.

    “Within-group competition for scarce resources reduces trust and trustworthiness (Charness, Masclet, & Villeval, 2013;Chowdhury & Gürtler, 2015;Liu, Lin, & Xin, 2014), decreases contributions to public goods (Barker & Barclay, 2016), diminishes cooperation (West et al., 2006), and hinders agreement in bargaining situations (Barclay & Stoller, 2014).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Barker & Barclay, 2016 source

      The General discussion reports that people are less willing to contribute money in a prisoner’s dilemma game under local competition. That supports reduced contributions in a related cooperative setting, but the paper does not establish decreased contributions to public goods specifically; its own experiments measured money spent harming others. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Contests and tournaments with rank-dependent financial rewards elicit more individual effort than piece-rate remuneration schemes.

    “Specifically, contests and tournaments, where financial rewards depend on rank performance, elicit more individual effort than piece-wise remuneration schemes and, in fact, drive individual effort expenditure beyond what is rational (Cason, Masters, & Sheremeta, 2010;Connelly, Tihanyi, Crook, & Gangloff, 2014;Dechenaux, Kovenock, & Sheremeta, 2015;Dickinson & Isaac, 1998).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Cason, Masters, & Sheremeta, 2010 source

      The Introduction and §2.1 describe rank-dependent contests and a piece-rate option. But §3.1 and the Conclusions compare achievement chiefly between proportional-prize and winner-take-all contests; they do not establish that contests elicit more individual effort than piece-rate pay. Section 3.3 reports excessive contest entry relative to expected payoffs, not effort expenditure beyond a rational level. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Connelly, Tihanyi, Crook, & Gangloff, 2014 source
    • cites Dechenaux, Kovenock, & Sheremeta, 2015 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Contests and tournaments drive individual effort expenditure beyond what is rational.

    “Specifically, contests and tournaments, where financial rewards depend on rank performance, elicit more individual effort than piece-wise remuneration schemes and, in fact, drive individual effort expenditure beyond what is rational (Cason, Masters, & Sheremeta, 2010;Connelly, Tihanyi, Crook, & Gangloff, 2014;Dechenaux, Kovenock, & Sheremeta, 2015;Dickinson & Isaac, 1998).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Cason, Masters, & Sheremeta, 2010 source

      The Introduction (p. unavailable) says the contests elicit effort, and Results §3.1 reports greater total achievement in proportional-prize than winner-take-all contests. But §3.1 says overall achievement is nearly identical when piece-rate effort is included, and §3.3 documents excessive contest entry, not individual effort expenditure beyond a rational level. The source therefore does not support the claimed excess individual effort or a general effort advantage over piece-rate pay. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Connelly, Tihanyi, Crook, & Gangloff, 2014 source
    • cites Dechenaux, Kovenock, & Sheremeta, 2015 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Within-group competition for scarce resources reduces trust and trustworthiness.

    “Within-group competition for scarce resources reduces trust and trustworthiness (Charness, Masclet, & Villeval, 2013;Chowdhury & Gürtler, 2015;Liu, Lin, & Xin, 2014), decreases contributions to public goods (Barker & Barclay, 2016), diminishes cooperation (West et al., 2006), and hinders agreement in bargaining situations (Barclay & Stoller, 2014).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Charness, Masclet, & Villeval, 2013

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chowdhury & Gürtler, 2015 source

      The discussion of sabotage in contests (section 4.1, p. 8) reports reduced productive effort and underreporting of rivals’ performance; the conclusion (p. 17) also notes that sabotage can occur in teams. Neither passage addresses whether within-group competition for scarce resources reduces trust or trustworthiness. Searches for those terms found no passage that does. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Liu, Lin, & Xin, 2014 source

      The Discussion reports that within-group competition reduced trustworthiness among acquaintances, but it improved trust rather than reducing it. The Abstract specifies that participants competed for financial rewards; it does not describe the rewards as scarce resources. (LLM, conf=0.92)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Within-group competition for scarce resources hinders agreement in bargaining situations.

    “Within-group competition for scarce resources reduces trust and trustworthiness (Charness, Masclet, & Villeval, 2013;Chowdhury & Gürtler, 2015;Liu, Lin, & Xin, 2014), decreases contributions to public goods (Barker & Barclay, 2016), diminishes cooperation (West et al., 2006), and hinders agreement in bargaining situations (Barclay & Stoller, 2014).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Barclay & Stoller, 2014 source

      The Results (section 3) report that participants under local competition demanded higher minimum offers in the ultimatum game, supporting greater rejection of unfair offers. That supports difficulty reaching agreement on low offers, but the paper does not establish that local competition reduced bargaining agreements overall; the claim is broader than the measured result. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Study design and conductown data

    Because pruned data are used, the treatment effect cannot be claimed for whole population; estimand is local average treatment effect for treated.

    “Since we are using pruned data, we cannot claim to measure the treatment effect on the whole population.Rather, the estimand in our statistical models is the local average treatment effect for the treated (Iacus et al., 2012).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Iacus et al., 2012 source

      Section 6.3.3 defines the estimand after matching as the average treatment effect for the matched people, which the paper calls a “local SATT.” But Section 2.3 says pruning only control units leaves the target quantity unchanged; the shift to a local estimand depends on pruning treated units, not merely on using pruned data. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Non-balanced design with non-independent observations is accounted for by fitting mixed effects models with random intercepts for strata, challenges, and individuals.

    “Essentially, we have a non-balanced design with non-independent observations and we account for it by fitting mixed effects models with random intercepts for strata, challenges, and individuals (Barr, Levy, Scheepers, & Tily, 2013).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Barr, Levy, Scheepers, & Tily, 2013 source

      In “Random effects in LMEMs and ANOVA,” the paper explains that random intercepts can account for clustering and non-independence across subjects and items. It does not prescribe intercepts for the citing study’s particular strata, challenges, and individuals. In “Identifying the maximal random effects structure,” it also advises trying random slopes for unbalanced within-unit designs, so intercepts alone are not generally sufficient. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Curved grading has been found to raise average effort in certain settings.

    “Grading on a curve", whereby course grades are determined in reference to others' performance, has been shown to increase average effort under some conditions (Andreoni & Brownback, 2017;Becker & Rosen, 1992)”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Andreoni & Brownback, 2017 source
    • cites Becker & Rosen, 1992 source

  • May combine two claims.

    Findingown data

    In Topcoder contests where submitted code is scored relative to other submissions rather than on an absolute scale, top performers increase effort while bottom performers decrease effort.

    “We find that in contests where the submitted code is evaluated against others' submissions, rather than using an absolute scale, top performers increase their effort while bottom performers decrease it.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared to average contestant in same challenge, the subjects are more experienced and higher rated, make more submissions, and perform better.

    “Compared to the average contestant in the same challenge, our "subjects" are more experienced and higher rated; further, they make more submissions and perform better (Table 5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Under relative scoring, low-rated coders' performance seems more predictable, whereas those in the 60-80 rating percentile seem less predictable.

    “It appears that relatively scored challenges make the performance of low-rated individuals more predictable, while that of individuals in the 60-80 percentile by rating less predictable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Performance predictability tends to be higher at both the low and high ends of the rating scale under relative scoring.

    “the U-shaped relationship in the right panel in Figure 2 suggests a tendency for higher predictability at the lower and higher ends of rating.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cyclomatic complexity typically increases with subsequent submissions, supporting the idea that it reflects elaboration and development rather than consolidation and refinement.

    “Relevantly to this, we find that the cyclomatic complexity typically increases with subsequent submissions, further supporting the idea that it reflects elaboration and development, rather than consolidation and refinement (Fig. 3, left).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    After matching and identifying individuals with rating who have at least one treatment and one control observation in a stratum, sample is 1,239 observations of 319 coders over 42 challenges in 12 strata; these coders made 12,914 successful code submissions.

    “After matching the challenges and identifying individuals with Topcoder rating who have at least one treatment and one control observation in one of the strata, we are left with 1,239 observations of 319 coders over 42 challenges in 12 strata; these coders made 12,914 successful code submissions altogether.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After matching, dataset has 52 challenges in 16 strata, with reasonable covariate balance for six coarsened variables.

    “After performing the matching procedure, we obtain a dataset of 52 challenges in 16 strata, with a reasonable covariate balance for the six coarsened variables (Fig. 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Findings apply only to experienced coders, not to first-time contestants.

    “In supplementary analyses, we found out that our results concern only coders with prior experience on the platform and do not extend to first-time contestants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among 31 relative-scoring challenges, six used downward comparison based on competitors beaten; the rest used upward comparison normalized by the best score.

    “Out of the 31 relatively scored challenges we analyze, in six the score depends on the number of competitors the player beats (downward comparison) while in the rest, the score is normalized by the current best score (upward comparison).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Among first-time contestants, relative scoring is associated with lower code complexity in the first submission (coefficient −0.441, p<0.05).

    “Relative scoring | | 0.319 | | -0.353 * | | -0.441 *”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Splitting by comparison direction suggests downward comparison discourages bottom-ranked coders, while upward comparison encourages top-ranked coders.

    “When we split the analyses by the direction of comparison, we observe that downward comparison appears to be the one that demotivates those already at the bottom of the ranking, while upward comparison is the one that motivates those at the top (Fig. 6).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Relative scoring produces better outcomes for top performers but lower engagement for bottom performers.

    “As a result, relative scoring leads to better outcomes for those at the top but lower engagement for bottom performers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Humans have a powerful drive to gain accurate self-evaluations and often compare their opinions and skills with others.

    “According to the eminent social psychologist Leon Festinger, we have a powerful drive to gain accurate self-evaluations and we often achieve this by comparing our opinions and skills to those of others (Festinger, 1954).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Festinger, 1954 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Topcoder is the largest competitive software development portal and constitutes both a marketplace for software solutions and a social network community.

    “Established in 2001, Topcoder is the largest competitive software development portal that constitutes both a marketplace for software solutions and a social network community.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Archak, 2010 source
    • cites K. J. Boudreau, Lacetera, & Lakhani, 2011 source
    • cites K. Boudreau, Helfat, Lakhani, & Menietti, 2012 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Competition increases spiteful behavior and sabotage, making individuals more likely to harm others at personal cost to diminish opponents' performance and increase their own chances of winning.

    “Competition also increases spiteful behavior and sabotage, making individuals more likely to harm others at a personal cost in order to diminish their opponents' performance and increase their own chances of winning (Harbring & Irlenbusch, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Harbring & Irlenbusch, 2011 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Reducing experiential inequality increases individual productivity and job satisfaction and improves organizational performance.

    “More generally, reducing experiential inequality has been shown to increase individual productivity and job satisfaction and improve organizational performance (Bapuji, Ertug, & Shaw, 2020;Stainback, Tomaskovic-Devey, & Skaggs, 2010).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Bapuji, Ertug, & Shaw, 2020 source
    • cites Stainback, Tomaskovic-Devey, & Skaggs, 2010 source

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Social comparison could result in rich-get-richer and poor-get-poorer dynamics that widen existing ability differences and make performance more predictable.

    “This implies that social comparison could result in a rich-get-richer and poor-get-poorer dynamics that widen existing ability differences and make performance more predictable, going directly against Festinger's arguments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Two mechanisms from Festinger's arguments are empirically supported: selection into and out of contests, and sabotage of winners.

    “Two of these mechanisms are empirically supported: the tendency to select into and out of contests (Dohmen & Falk, 2011;Eriksson, Teyssier, & Villeval, 2009) and the tendency to sabotage winners in contests (Chen, 2003).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Dohmen & Falk, 2011 source
    • cites Eriksson, Teyssier, & Villeval, 2009 source
    • cites Chen, 2003 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Online citizen science initiatives, learning platforms, user-contribution communities, and crowd workplaces flourish when their members keep engaging and learning.

    “Online citizen science initiatives, learning platforms, user-contribution communities, and crowd workplaces flourish when their members keep engaging and learning (Kittur et al., 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kittur et al., 2013 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Contests can be successfully employed to spur innovation.

    “As a result, contests can be successfully employed to spur innovation (Wright, 1983).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wright, 1983

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Topcoder maintains an active community of hundreds of thousands of designers, developers, data scientists, and programmers who regularly compete in programming challenges.

    “Topcoder maintains an active community of hundreds of thousands of designers, developers, data scientists, and programmers who regularly compete in programming challenges (Archak, 2010; K. J. Boudreau, Lacetera, & Lakhani, 2011;K. Boudreau, Helfat, Lakhani, & Menietti, 2012;Lakhani et al., 2013).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Archak, 2010 source
    • cites K. J. Boudreau, Lacetera, & Lakhani, 2011 source
    • cites K. Boudreau, Helfat, Lakhani, & Menietti, 2012 source

  • May combine two claims.

    Interpretationreasoning

    Social comparison may have differing effects on different people: some become more competitive, while others lose interest and decrease effort as their chances of winning drop.

    “In essence, social comparison may have differing effects on different people: some may become more competitive, while others may lose interest and decrease effort as their chances of winning drop.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Inequality in heterogeneous populations has two distinct aspects: dispersion of outcomes and predictability of outcomes.

    “When considering inequality in heterogeneous populations, there are two distinct aspects to take into account-the dispersion of outcomes and the predictability of outcomes (Lynn, Podolny, & Tao, 2009;Salganik, Dodds, & Watts, 2006).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Lynn, Podolny, & Tao, 2009 source
    • cites Salganik, Dodds, & Watts, 2006 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Game design elements such as likes, badges, skill levels, and leaderboards are used to increase learning in online courses, exercise on e-health platforms, or performance in standard organizational tasks.

    “This is the idea behind using game design elements such as likes, badges, skill levels, and leaderboards to increase learning in online courses, exercise on e-health platforms, or performance in standard organizational tasks (Denny, McDonald, Empson, Kelly, & Petersen, 2018;Koivisto & Hamari, 2019;Landers, Bauer, & Callan, 2017).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Denny, McDonald, Empson, Kelly, & Petersen, 2018 source
    • cites Koivisto & Hamari, 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Landers, Bauer, & Callan, 2017 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Social comparison can induce competitive behavior and increase effort even without financial incentives.

    “Still, social comparison can induce competitive behavior and increase effort even without financial incentives (Mago, Samak, & Sheremeta, 2016;Messick & McClintock, 1968;Sheremeta, 2010).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Mago, Samak, & Sheremeta, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Messick & McClintock, 1968 source
    • cites Sheremeta, 2010 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Study design and conductreasoning

    To measure performance quality, an objective metric independent of scoring type is required.

    “To measure the quality of performance, we require an objective metric that does not depend on the scoring type.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Cyclomatic complexity is a software complexity measure based on the decision structure of code.

    “Cyclomatic complexity is a software complexity measure that is based on the decision structure of the code (McCabe, 1976).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites McCabe, 1976 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Coders contribute to such challenges to learn and practice and to establish reputation in the community.

    “Coders contribute to such challenges in order to learn and practice and in order to establish their reputation in the community (K.J. Boudreau et al., 2011;K. Boudreau et al., 2012).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites K.J. Boudreau et al., 2011 source
    • cites K. Boudreau et al., 2012 source

  • May combine two claims.

    Study design and conductown data

    In most cases, observations in matched data span full range of values, giving enough variability to make conclusions about a diverse set of individuals on Topcoder.

    “Nevertheless, in most cases, the observations in the matched data span the full range of values, giving us enough variability to make conclusions about a diverse set of individuals on Topcoder.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The interaction term between scoring type and coder rating helps disentangle different effects expected on high-performers and low-performers.

    “The interaction term between scoring type and coder rating b 3 X j Z ij helps to disentangle the different effects we expect on high-performers and low-performers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Models control for treatment ordering effects by including number of challenges the coder has already participated in.

    “Additionally, we control for treatment ordering effects by including the number of challenges the coder has already participated in, which is another challenge-varying individual-level covariate (P ij ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The decision to model relationships as quadratic/cubic is informed by shape of bivariate relations in Figures 1 and 2 and best model fit.

    “In practice, we model the relationship between rating and proximity to rating as quadratic and the relationship between rating and code complexity, number of code lines, number of submissions, and time in challenge as cubic.This decision is informed by the shape of the bivariate relations depicted in Figure 1 and Figure 2 and the best model fit.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Gamma regression with log link is used for number of code lines, number of submissions, and code complexity because they are strictly positive, right-skewed, continuous count-like variables.

    “The outcome variables number of code lines written, number of submissions, and code complexity are strictly positive and can be seen as a continuous representation of count variables.The variables are heavily skewed to the right and hence, we fit gamma regression models with a log link function for these measures (Brooks et al., 2017;Faraway, 2005).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Brooks et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Faraway, 2005

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The rating measure is noisier for coders who have participated in fewer previous challenges.

    “It should be noted that this measure is not perfect since it is noisier for coders who have participated in fewer previous challenges on the platform.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Because data are observational, a quasi-experimental design is used to establish a causal link between scoring type and dispersion of effort, dispersion of performance, and predictability of performance.

    “Since our data are observational, we use a quasi-experimental research design to establish a causal link between scoring type and dispersion of effort, dispersion of performance, and predictability of performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There are two forms of social comparison: upward and downward.

    “There are two forms of social comparison people can make: upward and downward (Fiske, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fiske, 2011

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    To confirm robustness of effort findings, the paper also examines number of submissions and duration of participation as proportion of challenge duration.

    “To confirm the robustness of the findings on effort, we additionally look at the number of submissions and the duration of the coder's participation in the challenge, expressed as a proportion of the challenge duration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Observations are weighted by strata because not all types of challenges are equally represented.

    “Since not all types of challenges are equally represented, we additionally weigh the observations by strata.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The matched challenges are designed and posted by Topcoder to keep their user community stimulated and engaged.

    “This means that they are challenges designed and posted by Topcoder with the purpose to keep their user community stimulated and engaged.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The crossover experimental design compares difference in performance between treatment and control within individuals, ignoring individual differences in coding style and focusing on change in code complexity for each individual.

    “Most importantly, the cross-over experimental design we employ compares the difference in performance between the treatment and control within individuals.Essentially, we ignore individual differences in coding style and only focus on the change in the code complexity for each individual.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    These subjective experiences have been linked to short-sightedness, emphasis on small immediate rewards, and more risky behavior.

    “These subjective experiences have been linked to short-sightedness, emphasis on small and immediate rewards compared to large and delayed ones (Callan, Shead, & Olson, 2011), and more risky behavior (Payne, 2017).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Callan, Shead, & Olson, 2011 source
    • cites Payne, 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two cases with more than 10,000 new code lines between submissions were hard-coded data and were removed as outliers.

    “There are two cases where more than 10,000 new code lines were inserted between subsequent submissions, which on closer inspection turned out to constitute hard-coded data; we remove these two outliers from the relevant analyses.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Matching variables chosen are all variables that might be correlated with scoring type and outcomes: size, duration, difficulty, reward structure, and task.

    “We choose to match on all variables that might be correlated with the scoring type X and any of the outcomes Y : variables related to the size, duration, difficulty, reward structure, and task of the challenge.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Upward social comparison often triggers negative self-evaluation, resentment, and envy, but can also be motivating and self-enhancing.

    “Upward social comparison occurs when people compare themselves to those who are better off and although it often triggers negative self-evaluation, resentment, and envy, it can also be motivating and self-enhancing (Collins, 1996).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Collins, 1996 source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Topcoder Marathon Match rating is used as proxy for prior performance/ability.

    “For prior performance, which we use as proxy for ability, we use the coder's official Topcoder Marathon Match rating.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Topcoder Marathon Match rating is a valid proxy for coder ability/prior performance.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Beta regression is used for proximity to rating and time in challenge because they are positive and bounded by 1.

    “The outcome variables proximity to rating and time in challenge are also positive but bounded by 1 and hence, we fit a beta regression model for these two cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Lines of code is a common metric for programmer productivity and effort in software development management.

    “Our primary measure of effort is lines of code (LOC)-a common metric for programmer productivity and effort employed in software development management (Fenton & Neil, 2000).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fenton & Neil, 2000 source

  • May combine two claims.

    Premisecitation

    Downward social comparison generally boosts self-confidence and improves self-evaluation but may also induce negative affect, especially among those with low self-confidence.

    “Downward social comparison occurs when people compare themselves to those who are worse off and, in parallel, it generally boosts self-confidence and improves self-evaluation but may also induce negative affect, especially among those with low self-confidence (Buunk, Collins, Taylor, VanYperen, & Dakof, 1990).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Buunk, Collins, Taylor, VanYperen, & Dakof, 1990 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Cyclomatic complexity increases with modularization and design effort, but very convoluted and inefficient code would also have high cyclomatic complexity.

    “Although the measure increases with modularization and design effort (Shepperd, 1988), very convoluted and inefficient code would also have high cyclomatic complexity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Shepperd, 1988 source

  • May combine two claims.

    Study design and conductown data

    The correlation with rank percentile can be quite low in some cases, indicating the measure mainly captures elaboration and development of code rather than how well it addresses the problem.

    “Nevertheless, the correlation with rank percentile can be quite low in some cases (Fig. 3, right), which brings attention to the fact that the measure mainly captures the elaboration and development of the code, rather than how well it addresses the particular problem.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Data are partially crossed because individuals are nested within different challenges but not all individuals are present in all challenges.

    “Our data are partially crossed since individuals are nested within different challenges but not all individuals are present in all of the challenges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Social comparison exaggerates small differences in ability and effort, but individuals' reactions to ongoing results can feed back into behavior and reinforce these differences.

    “In short, social comparison exaggerates small differences in ability and effort but individuals' reaction to the ongoing results could feedback into individuals' behavior in a way that reinforces these differences.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Cyclomatic complexity measures how advanced and sophisticated the coder's solution is in terms of control logic.

    “We use cyclomatic complexity to measure how advanced and sophisticated the coder's solution is in terms of control logic (Munson & Khoshgoftaar, 1989).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Munson & Khoshgoftaar, 1989

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The limitation of noisier rating for less experienced coders is mitigated by controlling for number of previous challenges in statistical models.

    “We mitigate for this limitation, however, by controlling for the number of previous challenges in the statistical models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The LOC measure generally captures the amount of work and attention involved, though it does not differentiate small edits from major overhauls.

    “This measure does not differentiate between small edits, such as changing a variable name, and a major overhaul, such as rewriting the algorithm, but it generally captures the amount of work and attention involved.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Absolute final scores are deliberately scaled for relatively scored challenges, and final rank percentiles are influenced by skill-based selection into differently scored challenges, so neither can be used as an objective performance metric.

    “This precludes using the absolute final scores since they are deliberately scaled for relatively scored challenges, as well as the final rank percentiles since they are influenced by differences in skill-based selection into differently scored challenges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Leaderboards and similar feedback can create short-term client value yet also push away users, altering a community's size, makeup, and identity.

    “Leaderboards, rankings, and other-based feedback may help elicit value for clients in the short term but may also drive away some users, inadvertently impacting the size, composition, and identity of the community.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Without chances to cooperate, social comparison can lead lower achievers to try less hard, not more, thereby increasing performance inequality.

    “We found that, in the absence of opportunities for cooperation, social comparison can actually make underachievers exert less effort, not more, and thus produce more unequal performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The final matched set contained 52 challenges drawn from 49 unique problems.

    “The final set of 52 matched challenges corresponds to 49 unique problems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Missing values arose in cyclomatic complexity and file diff analyses when algorithms failed on submitted code.

    “The analyses involving cyclomatic complexity and file diffs also contain a number of missing values due to the algorithms failing to run on the submitted code.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Only contestants with at least one non-zero submission were included.

    “Similarly, when analyzing contestants in a challenge, we only considered individuals who made at least one non-zero submission.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    McCabe's cyclomatic complexity was used to measure code complexity.

    “The complexity of the source code was measured using cyclomatic complexity, as introduced by Thomas J. McCabe (McCabe, 1976).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites McCabe, 1976 source

  • May combine two claims.

    Interpretationcitation

    Social comparison, when paired with cooperation and idea sharing, can support learning, community, and innovation.

    “When combined with opportunities for cooperation and idea exchange, social comparison can in fact foster learning, community building, and innovation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Effort was quantified by summing newly inserted code lines across submissions.

    “To quantify effort, we estimate the total number of lines the coder wrote for the challenge.To do this, we sum over the number of new code lines inserted with every submission.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Matching and a quasi-experimental crossover design meant most data were discarded and the claims' scope was reduced.

    “To provide more accurate causal estimates, we used matching and a quasiexperimental crossover design.However, this made us disregard the majority of our data and narrow down the scope of our claims.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Five challenges with fewer than 10 contestants were also removed.

    “We additionally removed any challenge that had fewer than 10 contestants (5 challenges).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    SimpleDiff, Paul Butler's implementation using the Ratcliff/Obershelp algorithm, was the diff tool.

    “The particular implementation of the diff utility we used (SimpleDiff) was developed by Paul Butler and relies on the Ratcliff/Obershelp algorithm to solve the longest common subsequence problem in order to determine the smallest set of deletions and insertions to create one set from the other (Ratcliff & Metzener, 1988).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ratcliff & Metzener, 1988

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Study design and conductreasoning

    Rating analyses used 42 rather than 52 challenges because new contestants have no Topcoder rating.

    “Since challenges that took place soon after Topcoder started contain mainly new platform members, the number of observations that we end up analyzing when rating is involved is actually lower: 42 instead of the 52 challenges we originally matched.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Because an objective performance measure uncorrelated with scoring type was needed, official challenge metrics could not be used.

    “We required an objective measure of performance that is not correlated with the scoring type; we thus could not rely on the official performance metrics employed by the challenges.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The effort and performance models weight observations by stratum, adjust for treatment ordering via the count of prior challenges, and include random intercepts for individual, challenge, and stratum.

    “The models include weights by stratum and control for treatment ordering effects using the number of previous challenges the coder has participated in.The models also include random intercepts by individual, challenge, and stratum.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The issue is not social comparison itself but how it is used to intensify competition.

    “Yet, it is not social comparison per se that is the problem but its use to heighten competition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The Python package Lizard computed cyclomatic complexity.

    “We calculated cyclomatic complexity with the Python package Lizard (Yin, 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yin, 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Competition and social comparison may harm motivation and learning.

    “social comparison and competition can negatively impact motivation and learning”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The design handles self-selection bias well but is limited regarding sample selection bias.

    “Further, our research design is powerful when it comes to dealing with self-selection bias, but has limitations with respect to sample selection bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The study tested Festinger's effort prediction in competitive settings and found it incorrect.

    “Thus, we directly addressed Festinger's prediction on effort and proved it wrong in competitive settings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Performance-quality findings should be read only as solution elaboration and development, not as maintainability, optimization, or legibility.

    “We thus caution that our results on the quality of performance should be interpreted more narrowly in terms of the extent of elaboration and development of the solution, and not in reference to other possible criteria such as software maintainability, optimization, code legibility, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Interpretationown data

    Upward and downward social comparison may produce different effects.

    “the effects could deffer for upward and downward social comparison.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The unequal effort and performance effects may, over time, create a self-reinforcing cycle that widens the gap between bottom and top performers.

    “Nevertheless, over time, the unequal effects on effort and performance we discovered could lead to a self-reinforcing loop that solidifies and even increases differences between bottom and top performers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Unmeasured differences between relative and absolute scoring challenges could account for the effects.

    “it is still possible that relatively and absolutely scored challenges differ in respects that we have not captured and that may explain the observed effects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Topcoder does not represent other crowdsourcing platforms or online communities broadly.

    “We recognize that the Topcoder community is not representative of other crowdsourcing platforms or online communities in general.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    After removing 111 single-problem challenges, 256 challenges covering 214 unique problems remained.

    “we excluded 111 challenges that were based on a single problem (these presented a controlled experiment on the platform), leaving us with 256 challenges based on 214 unique problems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Halving the data after splitting prevents quantitative separation of upward and downward comparison effects.

    “Because we halve the data after the split, we cannot disentangle these effects quantitatively”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Cyclomatic complexity of the submitted source code is a suitable performance measure, since it tends to rise across successive submissions and correlates with rank percentile by final score.

    “The cyclomatic complexity of the source code is a suitable measure of performance because it generally increases in subsequent submissions (left) and is correlated with rank percentile by final score in the challenge (right).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Relative feedback can energize experienced high performers while discouraging learners.

    “While relative feedback may stimulate experienced overachievers, it may demotivate learners.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Social comparison raises effort among higher performers but lowers it among lower performers.

    “we presented evidence that social comparison increases effort for better-performing individuals but has exactly the opposite effects on worse-performing individuals.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Results may not generalize to contests with monetary prizes.

    “our findings do not necessarily transfer to contests that offer monetary rewards.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Public submissions with peer comments and evaluation in innovation contests help participants learn from others and enhance their own solutions.

    “innovation contests that make submissions public and allow for peer comments and evaluation help contestants learn and incorporate ideas from others' solutions and improve the quality of their own solution”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Bockstedt et al., 2016 source
    • cites Bullinger et al., 2010 source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Zero-point submissions were excluded as frequent and likely caused by minor bugs rather than effort.

    “When analyzing individual submissions we ignored 0-point submissions.These occur extremely often in the data and could be explained with minor bugs in the code that break the provisional scoring system.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Scoring-type labels came from Topcoder users in a dedicated platform challenge.

    “The classification by scoring type was provided by Topcoder users in a dedicated challenge on the platform.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Low-rated coders exert less effort and submit less complex solutions in relatively scored challenges involving downward social comparison, whereas high-rated coders exert more effort and submit more complex solutions in relatively scored challenges involving upward social comparison.

    “Low-rated coders put less effort and produce solutions with lower cyclomatic complexity in relatively scored challenges with downward social comparison (left), while high-rated coders put more effort and produce solutions with higher cyclomatic complexity in relatively scored challenges with upward social comparison (right).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Final outcomes stem from social comparison dynamics rather than from coders' prior expectations or dispositions.

    “indicating that the final outcomes are not driven by the contestants' a priori expectations and dispositions but are the result of social comparison dynamics.”
    Quote in the paperSubstantial claim
99 of 99

  • The authors state this as an assumption.

    Social comparison is more prominent in relatively scored challenges than in absolutely scored ones.

    “We assume that social comparison is more prominent in the relatively scored challenges compared to the absolutely scored ones.”

  • The paper relies on this without support.

    In creative tasks, effort is positively correlated with but distinct from performance.

    “In creative tasks, input, or effort, is positively correlated with but still distinct from output, or performance.”

  • The paper relies on this without support.

    Previous literature has not directly studied the effects of social comparison on inequality in effort and performance.

    “Previous literature has not directly studied the effects of social comparison on inequality in effort and performance but it nevertheless contains some insights and suggestive empirical evidence, albeit contradictory.”

  • The paper relies on this without support.

    When individual achievement is measured and evaluated in reference to others, success becomes a moving target.

    “When individual achievement is measured and evaluated in reference to others, success becomes a moving target.”

  • The paper relies on this without support.

    The Topcoder Marathon Match rating is a valid proxy for coder ability/prior performance.

    “For prior performance, which we use as proxy for ability, we use the coder's official Topcoder Marathon Match rating.”

  • The authors state this as an assumption.

    Relative and absolute scoring challenges differ solely in their emphasis on social comparison.

    “We assumed that the only difference between the two types of challenges is the emphasis on social comparison but we cannot entirely preclude other subtle differences responsible for alternative behavioral mechanisms.”

  • The authors state this as an assumption.

    Newcomers select their initial challenge at random.

    “Assuming that first-timers randomly choose their first challenge, we tested if the performance, effort, and likelihood to participate again are different for those who happened to enter Topcoder with a relatively scored challenge versus absolutely scored challenge.”

  • The paper relies on this without support.

    The scoring-type labels provided by Topcoder users accurately reflect the actual scoring type.

    “The classification by scoring type was provided by Topcoder users in a dedicated challenge on the platform.”

  • The paper relies on this without support.

    Cyclomatic complexity validly measures software quality in terms of elaborated logical flow and development.

    “We chose McCabe's cyclomatic complexity with the intention to quantify software quality in terms of its elaborated logical flow and development.”
9 of 9

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Nikolay Archak and Arun Sundararajan . 2009 . Optimal design of crowdsourcing contests . ICIS 2009 Proceedings , Vol. 200 (2009), 1--16. Nikolay Archak and Arun Sundararajan. 2009. Optimal design of crowdsourcing contests. ICIS 2009 Proceedings, Vol. 200 (2009), 1--16.
  2. Julian J. Faraway . 2005. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models 1 edition ed.) . Chapman and Hall/CRC , Boca Raton . Julian J. Faraway. 2005. Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models 1 edition ed.). Chapman and Hall/CRC, Boca Raton.
  3. Jeff Howe . 2008 . Crowdsourcing: How the Power of the Crowd is Driving the Future of Business . Random House, London , UK. Jeff Howe. 2008. Crowdsourcing: How the Power of the Crowd is Driving the Future of Business. Random House, London, UK.
  4. James Wayne Hunt and M. Douglas MacIlroy . 1976. An Algorithm for Differential File Comparison. Bell Laboratories , Murray Hill, NJ . James Wayne Hunt and M. Douglas MacIlroy. 1976. An Algorithm for Differential File Comparison. Bell Laboratories, Murray Hill, NJ.
  5. B. Keith Payne . 2017. The Broken Ladder: How Inequality Affects the Way We Think, Live, and Die . Viking , New York . B. Keith Payne. 2017. The Broken Ladder: How Inequality Affects the Way We Think, Live, and Die. Viking, New York.
  6. John W. Ratcliff and David E. Metzener . 1988 . Pattern-matching-the gestalt approach . Dr Dobbs Journal , Vol. 13 , 7 (1988), 46 . John W. Ratcliff and David E. Metzener. 1988. Pattern-matching-the gestalt approach. Dr Dobbs Journal, Vol. 13, 7 (1988), 46.
  7. Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets . In Fifth International AAAI Conference on Weblogs and Social Media. Association for the Advancement of Artificial Intelligence , Palo Alto, CA, 321--328. Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Fifth International AAAI Conference on Weblogs and Social Media. Association for the Advancement of Artificial Intelligence, Palo Alto, CA, 321--328.

Cited by

Sorted by citations

Related

Authors
Tsvetkova, M., Müller, S., Vuculescu, O., Ham, H., & Sergeev, R. A.
Affiliations
  • London School of Economics and Political Science (GB)
  • Aarhus University (DK)
  • Harvard University (US)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2022-11-07
Volume
6
Issue
CSCW2
Pages
1-27
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3555649
OpenAlex
W4309618420
Semantic Scholar
2a74399c474849148a628bd21681b6446cbc9b24
Cited by
8
References
79
Citations by year
2026: 1 · 2025: 4 · 2024: 1 · 2023: 1 · 2022: 1
FWCI
1.43
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (acceptedVersion)
Funders
Volkswagen Foundation (92 173)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
Keywords
social comparison, crowdsourcing contests, ability differences, gamification, online communities, TopCoder, programming competition, user engagement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review