Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 101–200 of 1,372
Machery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 110
Z-curve 2.0: Estimating Replication Rates and Discovery Rates
Bartos̆, F., & Schimmack, U. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2720 · Cited by 126
Reference-Dependent Preferences
O’Donoghue, T., & Sprenger, C. D. (2018) · Handbook of behavioral economics · doi:10.1016/bs.hesbe.2018.07.003 · Cited by 176
Individual discount rates: a meta-analysis of experimental evidence
Matousek, J., Havranek, T., & Irsova, Z. (2022) · Experimental Economics · doi:10.1007/s10683-021-09716-9 · Cited by 84
A Unified Welfare Analysis of Government Policies
Hendren, N., & Sprung-Keyser, B. (2019) · doi:10.3386/w26144 · Cited by 88
“To place these estimates in perspective, Andrews and Kasy (Forthcoming) use data from Camerer et al. (2016) to estimate that lab experiments in economics are 26-35 times more likely to be published if they have statistically significant results; in contrast, they estimate that studies of the minimum wage are 3 times m…”
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116
“This project was followed by other replication attempts in psychology [32], social sciences [28] and economics [29], with replication rates of 54%, 62% and 61%, respectively.”
MethodReproducibility in Neuroimaging Analysis: Challenges and Solutions
Botvinik‐Nezer, R., & Wager, T. D. (2022) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2022.12.006 · Cited by 116
Reproducibility of clinical research in critical care: a scoping review
Niven, D. J., McCormick, T. J., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., Barnes, T., & Stelfox, H. T. (2018) · BMC Medicine · doi:10.1186/s12916-018-1018-6 · Cited by 91
“[18] found that, for 18 experimental economic studies, 11 (61%) reproduction attempts found a significant effect in the same direction as the original study.”
BackgroundSchaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87
“Previous research has demonstrated that academics sometimes perform well in anticipating the results of scientific studies, based on limited information such as article abstracts and study materials (Camerer et al., 2016; Dreber et al., 2015; Forsell et al., 2019).”
Self-citationManiadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
“What does a quantitative meta-research study have to offer? First, often expert opinion might not be enough, as evidenced by the fact that in Camerer et al.’s (2016) study, the predictive accuracy of experts about replicability did not outperform objective indices such as sample size and p-value. Moreover, Della Vigna…”
InfluentialDawid, H., Harting, P., van der Hoog, S., & Neugart, M. (2018) · Journal of Evolutionary Economics · doi:10.1007/s00191-018-0594-0 · Cited by 89
“In this respect, it has been shown that macroeconomic patterns 5For example, Camerer et al. (2016) report that for only 11 out of 18 experimental results published in top five economics journals the results could be independently reproduced by third parties. emerging from the modelled micro-level…”
Making Replication MainstreamPreprint
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, B. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4tg9c · Cited by 92
Bhanot, S. P. (2018) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.11.002 · Cited by 111
“This can be thought of as an “ostrich effect” response to injunctive norms, whereby people “stick their head in the 12 sand” via information avoidance when they suspect future information will reflect negatively on them (Eil & Rao, 2011; Karlsson, Loewenstein, & Seppi, 2009; Sicherman, Loewenstein, Seppi, & Utkus, 201…”
BackgroundMethodInfluentialAn investigation of irreproducibility in maximum likelihood phylogenetic inference
Shen, X., Li, Y., Hittinger, C. T., Chen, X., & Rokas, A. (2020) · Nature Communications · doi:10.1038/s41467-020-20005-6 · Cited by 93
Preferences and constraints: the value of economic games for studying human behaviour
Pisor, A. C., Gervais, M. M., Purzycki, B. G., & Ross, C. T. (2020) · Royal Society Open Science · doi:10.1098/rsos.192090 · Cited by 83
“7 : 192090 replication is one that finds effects in the same direction as those of the original experiment (see [87] for examples).”
Escaping Paternalism: Rationality, Behavioral Economics, and Public Policy
Rizzo, M. J., & Whitman, G. (2019) · Cambridge University Press eBooks · doi:10.1017/9781139061810 · Cited by 182
Many Labs 2: Investigating Variation in Replicability Across Sample and Setting
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J., Babalola, M. T., Bahník, Š., et al. (2018) · doi:10.31234/osf.io/9654g · Cited by 75
“…sciences, there is accumulating evidence that fewer published effects replicate than might be expected, and that replication effect sizes are typically smaller than original effect sizes (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014; Open Science Collaboration, 2015).”
BackgroundResultInfluentialA scientific approach to entrepreneurial decision‐making: Large‐scale replication and extension
Camuffo, A., Gambardella, A., Messinese, D., Novelli, E., Paolucci, E., & Spina, C. (2024) · Strategic Management Journal · doi:10.1002/smj.3580 · Cited by 84
Yang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T. H., Lagisz, M., & Nakagawa, S. (2023) · BMC Biology · doi:10.1186/s12915-022-01485-y · Cited by 119
“Although no single indicator can capture the true extent or all relevant evidence of the replication crisis [93], we have provided clear evidence that, as in many other disciplines [1, 2, 4], previously published findings in ecology and evolutionary biology are likely to have low replicability.”
BackgroundOpen Science Is Liberating and Can Foster Creativity
Frankenhuis, W. E., & Nettle, D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618767878 · Cited by 81
“Given these quite understandable tendencies, it should be no surprise that when studies in the behavioral and life sciences are replicated, we do not consistently see in the new results the patterns reported in the original studies (Begley & Ioannidis, 2015; Camerer et al., 2016; Frank et al., 2017; Klein et al., 2014…”
ResultAl‐Ubaydli, O., Lee, M. S., List, J. A., Mackevicius, C. L., & Suskind, D. L. (2020) · Behavioural Public Policy · doi:10.1017/bpp.2020.17 · Cited by 92
“This landscape change would be welcome given the current credibility crises in science (Jennions & Moller, 2001; Ioannidis, 2005; Bettis, 2012; Nosek et al., 2012, Camerer et al., 2016).”
BackgroundResearcher degrees of freedom in phonetic research
Roettger, T. B. (2019) · Laboratory Phonology Journal of the Association for Laboratory Phonology · doi:10.5334/labphon.147 · Cited by 80
Publication Bias and Editorial Statement on Negative Findings
Blanco-Perez, C., & Brodeur, A. (2020) · The Economic Journal · doi:10.1093/ej/ueaa011 · Cited by 76
An exercise in self-replication: Replicating Shah, Mullainathan, and Shafir (2012)
Shah, A. K., Mullainathan, S., & Shafir, E. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.12.001 · Cited by 72
“Much of the attention on the reproducibility of psychological science has been focused on large-scale efforts to examine how many findings replicate in a cross-section of studies (e.g., Camerer et al., 2016, 2018; Open Science Collaboration, 2015; Klein, 2014).”
BackgroundBaltussen, G., Swinkels, L. A. P., & van Vliet, P. (2019) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.06.030 · Cited by 92
Pickett, J. T., & Roche, S. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-017-9886-2 · Cited by 82
“Even when findings have replicated, the effect sizes have consistently been smaller than in the original studies (Camerer et al. 2016; Open Science Collaboration 2015).”
The role of replication studies in ecology
Fraser, H. S., Barnett, A., Parker, T. H., & Fidler, F. M. (2020) · Ecology and Evolution · doi:10.1002/ece3.6330 · Cited by 91
“…et al., 2018; Open Science Collaboration, 2015), from 11% to 49% in preclinical biomedicine (Freedman, Cockburn, & Simcoe, 2015), and from 67% to 78% in economics research (Camerer et al., 2016) depending on the study, and the measure of “successful” used (see Fidler et al., 2017 for a summary).”
MethodHendriks, F., Kienhues, D., & Bromme, R. (2020) · Public Understanding of Science · doi:10.1177/0963662520902383 · Cited by 73
brainlife.io: a decentralized and open-source cloud platform to support neuroscience research
Hayashi, S., Caron, B. A., Heinsfeld, A. S., Vinci‐Booher, S., McPherson, B., Bullock, D. N., Bertò, G., Niso, G., Hanekamp, S., Levitas, D. J., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02237-2 · Cited by 99
“…that limited sample sizes might have hindered the validity of early, foundational results in hypothesis-driven cognitive neuroscience research, 10–16 but reproducibility issues can be found in biological science, 17,18 psychology, 12 data science, and computational methods, 19,20 cancer…”
BackgroundLeuz, C. (2018) · National Bureau of Economic Research · doi:10.3386/w24535 · Cited by 75
The Generalizability of Online Experiments Conducted During The COVID-19 PandemicPreprint
Peyton, K., Huber, G. A., & Coppock, A. (2020) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/s45yg · Cited by 72
Heterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 103
Self-citationReproducibility in Accounting Research: Views of the Research Community
Hail, L., Lang, M. H., & Leuz, C. (2020) · Journal of Accounting Research · doi:10.1111/1475-679x.12305 · Cited by 70
Statistical analyses for studying replication: Meta-analytic perspectives.
Hedges, L. V., & Schauer, J. M. (2019) · Psychological Methods · doi:10.1037/met0000189 · Cited by 69
“Similar challenges have also emerged in psychology (e.g., Open Science Collaboration, 2015) and economics (e.g., Camerer et al., 2016).”
BackgroundDistinct roles of theta and alpha oscillations in the involuntary capture of goal-directed attention
Harris, A. M., Dux, P. E., Jones, C. N., & Mattingley, J. B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.008 · Cited by 73
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 66
“Across many disciplines, including medicine (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011), economics (Camerer et al., 2016; Chang & Li, 2017; McCullough, McGeary, & Harrison, 2006), psychology (Ebersole et al.”
BackgroundSelf-citationSystematizing Confidence in Open Research and Evidence (SCORE)Preprint
Alipourfard, N., Arendt, B., Benjamin, D. M., Benkler, N., Bishop, M. M., Burstein, M. H., Bush, M., Caverlee, J., Chen, Y., Clark, C., et al. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/46mnb · Cited by 93
Self-citationNadelhoffer, T., Shepard, J., Crone, D. L., Everett, J. A. C., Earp, B. D., & Levy, N. L. (2020) · Cognition · doi:10.1016/j.cognition.2020.104342 · Cited by 70
Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E.-J. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 64
“Notably, the optimistic perspective does not seem to be unique to laypeople; experts similarly overestimated replicability in investigations by Dreber et al. (2015), Camerer et al. (2016), and Forsell et al. (2018), though not the one by Camerer et al. (2018).”
Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66
“Large - scale studies have demonstrated limited replication of research in Psychology (24, 25), Economics (26), the Social Sciences more generally (27), and Medicine (28–30).”
Using Bayes factors to evaluate evidence for no effect: examples from the SIPS project
Dienes, Z., Coulton, S., & Heather, N. (2018) · Addiction · doi:10.1111/add.14002 · Cited by 65
Making ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 71
More Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 73
The quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 89
The Bull of Wall Street: Experimental Analysis of Testosterone and Asset Trading
Nadler, A., Jiao, P., Johnson, C. J., Alexander, V., & Zak, P. J. (2017) · Management Science · doi:10.1287/mnsc.2017.2836 · Cited by 85
“Despite promising early findings, the field of hormonal neuroeconomics is developing, and studies in this area may require replication and corroboration for robustness, as done in other areas of experimental economics (Camerer et al. 2016).”
Challenges in Building Usable Knowledge in Education
Hedges, L. V. (2018) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1375583 · Cited by 63
“In evaluating the evidence about replication, particularly about replication in the social sciences, it is important to recognize that there is no generally accepted definition of replication, something acknowledged by both the Open Science Collaborative (2016) and Camerer et al. (2016).”
BackgroundMethodOptimally generate policy-based evidence before scaling
List, J. A. (2024) · Nature · doi:10.1038/s41586-023-06972-y · Cited by 119
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 71
The Critique of Empirical Social Science: New Policies atManagement and Organization Review
Lewin, A. Y., Chiu, C., Fey, C. F., Levine, S. S., McDermott, G., Murmann, J. P., & Tsang, E. W. K. (2016) · Management and Organization Review · doi:10.1017/mor.2016.43 · Cited by 68
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan Approach
Burlig, F. (2016) · Economics Letters · doi:10.1016/j.econlet.2018.03.036 · Cited by 79
Exploring online customer experience: A systematic literature review and research agenda
Kacprzak, A., & Hensel, P. G. (2023) · International Journal of Consumer Studies · doi:10.1111/ijcs.12966 · Cited by 97
The past, present, and future of Registered ReportsPreprint
Chambers, C., & Tzavella, L. (2020) · doi:10.31222/osf.io/43298 · Cited by 63
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020) · Psychological Methods · doi:10.1037/met0000365 · Cited by 88
“…errors biased toward significance and inflated effect sizes potentially constitute an important factor in the observed disappearance and decline of effects in replication studies such as Open Science Collaboration (2015); Camerer et al. (2016); R. A. Klein et al. (2018), and Camerer et al. (2018).”
BackgroundGeneralizability is not optional: insights from a cross-cultural study of social discounting
Tiokhin, L., Hackman, J., Munira, S., Jesmin, K., & Hruschka, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181386 · Cited by 57
Alempaki, D., Canic, E., Mullett, T. L., Skylark, W. J., Starmer, C., Stewart, N., & Tufano, F. (2019) · Management Science · doi:10.1287/mnsc.2018.3170 · Cited by 61
“That question is pertinent, not least, in the light of contemporary controversy surrounding the replicability of many of the findings in the behavioural sciences and elsewhere (e.g., Maniadis et al. 2014, Open Science Collaboration 2015, Camerer et al. 2016).”
BackgroundYang, Y., Hillebrand, H., Lagisz, M., Cleasby, I. R., & Nakagawa, S. (2021) · Global Change Biology · doi:10.1111/gcb.15972 · Cited by 75
“Such studies are likely to call for more collaboration and team science (Camerer et al., 2016; O’Dea et al., 2021), and the use of large- scale ecosystem research infrastructures (Roy et al., 2021).”
BackgroundFecher, B., Friesike, S., Hebing, M., & Linek, S. B. (2017) · Humanities and Social Sciences Communications · doi:10.1057/palcomms.2017.51 · Cited by 67
A Critical Analysis of Databases used in Financial Misconduct ResearchPreprint
Karpoff, J. M., Koester, A., Lee, D. S., & Martin, G. S. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2112569 · Cited by 64
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 59
“The Reproducibility Project Psychology (RPP; [5]) and the Experimental Economics Replication Project (EE-RP; [6]) are two projects that studied the replicability of psychological and economic experimental research by replicating published research.”
InfluentialHensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 78
Improving preclinical studies through replications
Drude, N., Martinez‐Gamboa, L., Danziger, M., Dirnagl, U., & Toelch, U. (2021) · eLife · doi:10.7554/elife.62101 · Cited by 63
Closing a Mental Account: The Realization Effect for Gains and Losses
Merkle, C., Müller-Dethard, J., & Weber, M. (2018) · Experimental Economics · doi:10.1007/s10683-020-09663-x · Cited by 64
A new approach to data access and research transparency (DART)
Beugelsdijk, S., van Witteloostuijn, A. C., & Meyer, K. E. (2020) · Journal of International Business Studies · doi:10.1057/s41267-020-00323-z · Cited by 65
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 84
Self-citationRETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
RetractedTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 50
Reducing bias, increasing transparency, and calibrating confidence with preregistrationPreprint
Hardwicke, T. E., & Wagenmakers, E. (2021) · doi:10.31222/osf.io/d7bcu · Cited by 55
Design Bioethics: A Theoretical Framework and Argument for Innovation in Bioethics Research
Pavarini, G., McMillan, R., Robinson, A., & Singh, I. (2021) · The American Journal of Bioethics · doi:10.1080/15265161.2020.1863508 · Cited by 60
“Incentives might also be used to motivate players, as typically done in laboratory experiments in economics (Camerer et al. 2016).”
BackgroundA creative destruction approach to replication: Implicit work and sex morality across cultures
Tierney, W., Hardy, J., Ebersole, C. R., Viganola, D., Clemente, E. G., Gordon, M., Hoogeveen, S., Haaf, J., Dreber, A., Johannesson, M., et al. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104060 · Cited by 51
“psychology (Open Science Collaboration, 2015), 11–25% in biomedicine (Begley & Ellis, 2012; Prinz, Schlange, & Asadullah, 2011), 61% in experimental economics (Camerer et al., 2016), 70% in experimental philosophy (Cova et al.”
Self-citationPage, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 61
Rethinking delusions: A selective review of delusion research through a computational lens
Ashinoff, B. K., Singletary, N. M., Baker, S. C., & Horga, G. (2021) · Schizophrenia Research · doi:10.1016/j.schres.2021.01.023 · Cited by 64
“Incentive compatibility is thought to contribute to the high replicability of economics paradigms, encouraging the reporting of true preferences and beliefs (Camerer, 1997; Camerer and Mobbs, 2017; Camerer et al., 2016).”
BackgroundShould the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0003 · Cited by 64
Pownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z. M., Grinschgl, S., Elsherif, M. M., Gilligan-Lee, K. A., Oliveira, C. M., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221255 · Cited by 78
“animal behaviour [15,16], cancer biology [17], and economics [18]).”
BackgroundBehavioral Experiments in Health Economics
Galizzi, M. M., & Wiesen, D. (2018) · Oxford University Press eBooks · doi:10.1093/acrefore/9780190625979.013.244 · Cited by 68
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study
Botvinik‐Nezer, R., Iwanir, R., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Dreber, A., Camerer, C. F., Poldrack, R. A., & Schönberg, T. (2019) · Scientific Data · doi:10.1038/s41597-019-0113-7 · Cited by 53
Self-citationQuestionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 82
Peters, J., Fiala, N., & Neubauer, F. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.05.009 · Cited by 76
“Within economics, a growing recent literature raises concerns about the replicability of empirical findings (Brodeur et al., 2020, 2016; Camerer et al., 2016; Christensen and Miguel, 2018; Ferraro and Shukla, 2022, 2020; Huntington-Klein et al., 2021; Ioannidis et al., 2017; Peters et al., 2018;…”
Güttinger, S. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-019-0269-1 · Cited by 84
A wakeup call for the field of entrepreneurship and its evaluators
McMullen, J. S. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2019.02.004 · Cited by 51
Eitan, O., Viganola, D., Inbar, Y., Dreber, A., Johannesson, M., Pfeiffer, T., Thau, S., & Uhlmann, E. L. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.06.004 · Cited by 50
Self-citationThomas, C., Otis, N. G., Abraham, J., Markus, H. R., & Walton, G. M. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1917046117 · Cited by 64
“Much of the current evidence on such “forecasting” aims to estimate the potential for scientific replicability (40, 41).”
BackgroundOpen Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P., & Carneiro, C. F. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 49
Chhin, C. S., Taylor, K. A., & Wei, W. S. (2018) · Educational Researcher · doi:10.3102/0013189x18788047 · Cited by 49
“Across these diverse fields, a limited number of replication studies and low rates of reproducibility and replication are common themes (e.g., Begley & Ellis, 2012; Camerer et al., 2016; Makel, Plucker, & Hegarty, 2012; Prinz, Schlange, & Asadullah, 2011).”
BackgroundReporting errors and biases in published empirical findings: Evidence from innovation research
Bruns, S. B., Asanov, I., Bode, R., Dunger, M., Funk, C., Hassan, S. M., Hauschildt, J., Heinisch, D. P., Kempa, K., König, J., et al. (2019) · Research Policy · doi:10.1016/j.respol.2019.05.005 · Cited by 61
Wang, J., Iversen, T., Hennig‐Schmidt, H., & Godager, G. (2020) · European Economic Review · doi:10.1016/j.euroecorev.2020.103411 · Cited by 49
Mayo‐Wilson, E., Grant, S., Supplee, L. H., Kianersi, S., Amin, A., DeHaven, A., & Mellor, D. T. (2021) · Research Integrity and Peer Review · doi:10.1186/s41073-021-00112-8 · Cited by 57
“Evidence that much empirical research cannot be reproduced [2]—including basic experiments in psychology [3], economics [4], and the social sciences [5]—has led to concerns about responsible research conduct and a “reproducibility crisis” [6], with a consequent “credibility revolution” focused largely on increasing re…”
BackgroundRommel, J., Sagebiel, J., Baaken, M. C., et al. (2022) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13330 · Cited by 48
Ten simple rules for good research practice
Schwab, S., Janiaud, P., Dayan, M., Amrhein, V., Panczak, R., Palagi, P. M., Hemkens, L. G., Ramon, M., Rothen, N., Senn, S. J., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010139 · Cited by 55
False confidence, non-additive beliefs, and valid statistical inference
Martin, R. (2019) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2019.06.005 · Cited by 72
“Unfortunately, the support for his claim has continued to mount—see Nuzzo (2014) and the reports in Nature—and the gravity of the problem is now becoming clear: large-scale replication projects have found that just 37/97 = 39% of findings in psychology (Open Science Collaboration 2015), 11/18 = 61% in experimental eco…”
BackgroundComputational Reproducibility in The Wildlife Society's Flagship Journals
ArchMiller, A. A., Johnson, A. D., Nolan, J., Edwards, M., Elliott, L. H., Ferguson, J. M., Iannarilli, F., Vélez, J., Vitense, K., Johnson, D. H., et al. (2020) · Journal of Wildlife Management · doi:10.1002/jwmg.21855 · Cited by 45
“Within the last 10 years, researchers have reported glaring issues in replication in several scienti fi c fi elds, including cancer biology (Begley and Ellis 2012), experimental psychology (Open Science Framework 2015), economics (Camerer et al. 2016), and animal behavior (Wang et al. 2018).”
BackgroundGordon, M., Viganola, D., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Twardy, C. R., et al. (2020) · Royal Society Open Science · doi:10.1098/rsos.200566 · Cited by 54
Self-citationMede, N. G., Schäfer, M. S., Ziegler, R., & Weißkopf, M. (2020) · Public Understanding of Science · doi:10.1177/0963662520954370 · Cited by 63
“Poor reproducibility 1 rates were also found in medicine (Freedman et al., 2015), economics (Camerer et al., 2016), biology (Baker and Dolgin, 2017), and experimental social science (Camerer et al., 2018).”
Randomized Control Trials in the Field of Development: A Critical Perspective
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · doi:10.1093/oso/9780198865360.001.0001 · Cited by 88
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
Reproducibility in Management Science
Fišar, M., Greiner, B., Huber, C., Katok, E., & Özkes, A. (2023) · Management Science · doi:10.1287/mnsc.2023.03556 · Cited by 85
Enriching the CAP evaluation toolbox with experimental approaches: introduction to the special issue
Thoyer, S., & Préget, R. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz024 · Cited by 62
Gordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44
“The replications and the forecasting results were included in a single publication for EERP [6] and SSRP [7].”
BackgroundMethodInfluentialSelf-citationA Proposal to Organize and Promote Replications
Coffman, L. C., Niederle, M., & Wilson, A. J. (2017) · American Economic Review · doi:10.1257/aer.p20171122 · Cited by 50
“Moreover, in an attempt to replicate a set of recent experimental articles, Camerer et al. (2016) were able to successfully replicate 61 percent of the studies, substantially less than the original p -values would suggest.”
BackgroundResultInfluential
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.