Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 901–1,000 of 1,372
A personal view on traits useful for success in science: Daniel S. Lehrman Award Lecture
Blaustein, J. D. (2018) · Hormones and Behavior · doi:10.1016/j.yhbeh.2018.12.005 · Cited by 2
Challenges and Problems of Neuroeconomics: Several Tasks for Social Scientists
Müller, M. (2019) · Teorie vědy / Theory of Science · doi:10.46938/tv.2018.410 · Cited by 3
What’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2
Die Replikationskrise in der Psychologie
Spitzer, M., & Spitzer, M. (2020) · Nervenheilkunde · doi:10.1055/a-1095-0144 · Cited by 2
“Die Veröffentlichung von Originaldaten bewirkt, dass die Robustheit des berichteten Effekts durch andere unabhängige Gruppen überprüft werden kann [18, 78].”
BackgroundPavlíček, A., & Sudzina, F. (2020) · Sustainability · doi:10.3390/su12125162 · Cited by 4
“Because of the replication crisis in psychology and economics [42,43], it would be advisable to replicate the research both in the Czech Republic and in other countries.”
BackgroundPoverty, Irrationality, and the Value of Cash Transfers
Acland, D. J. (2021) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2020.22 · Cited by 2
“6 It should be noted that an attempt to replicate these results failed (Camerer et al., 2016). make smaller/sooner vs larger/later choices.”
BackgroundThe Paternalistic Turn in Behavioral Law and Economics: A Critique
Rizzo, M. J. (2021) · Review of Law & Economics · doi:10.1515/rle-2021-0056 · Cited by 2
Dilbilim Araştırmalarında Açık Bilim
ATAMAN, E., ÇAĞLAR, O. C., & Kırkıcı, B. (2021) · Dilbilim Araştırmaları Dergisi · doi:10.18492/dad.936072 · Cited by 4
Rigour and reproducibility in Canadian research: call for a coordinated approach
Thibault, R. T., Munafo, M. R., & Moher, D. (2022) · FACETS · doi:10.1139/facets-2021-0162 · Cited by 3
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2
Two Person Bargaining Mechanisms: A Laboratory Experiment
Bol, D., Laslier, J., & Núñez, M. (2022) · Group Decision and Negotiation · doi:10.1007/s10726-022-09793-y · Cited by 2
Publish without bias or perish without replications
Ventura, R. (2022) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2022.08.010 · Cited by 2
Rajčáni, J., Vargová, L., Adamkovic, M., & Kačmár, P. (2023) · Studia Psychologica · doi:10.31577/sp.2023.01.866 · Cited by 3
Open Science Practices in The Journal Human Factors : 2017–2022
McCarley, J. S., Rose, L. E., Fischer, S., Haney, M. S., Hyk, A., Kuhn, K., Meredith, B., Moussaoui, J. R., Andrade, F. M. G., Pietrok, E. M., et al. (2023) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/21695067231192276 · Cited by 4
Forecasting long-run causal effects
Bernard, D. R., Himmelstein, M., Karger, E., & Schoenegger, P. (2026) · Journal of Development Economics · doi:10.1016/j.jdeveco.2026.103820 · Cited by 2
Experimental approach to development economics: a review of issues and options
Sekhar, C. S. C., & Thapa, N. (2024) · Journal of Economic Methodology · doi:10.1080/1350178x.2024.2341292 · Cited by 5
“Camerer et al. (2016) found that the size of the e ff ect in replications of RCTs was, on an average, one-third lower than the original in 18 laboratory experiments published in the AER and QJE .”
ResultCrisis, Confidence, and the Limits of Replication
Brown, J. (2024) · Zygon® · doi:10.16995/zygon.11502 · Cited by 3
TreeRing: A GameSafe parser for z-Tree
Jiang, M., & Li, J. (2019) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2019.02.001 · Cited by 2
Improving the Reliability and Generalizability of Scientific Research
Tiokhin, L. (2018) · Cited by 2
Trust and economic growth: Is there a relation?
Borisova, E., Bryzgalin, V. A., & Levina, I. (2020) · Voprosy Ekonomiki · doi:10.32609/0042-8736-2020-10-68-82 · Cited by 4
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
First Principles, Fallibilism, and EconomicsPreprint
Hoover, K. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2901539 · Cited by 2
Dissertation R.C.M. van AertPreprint
van Aert, R. C. M. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/eqhjd · Cited by 3
The sceptical Bayes factor for the assessment of replication successPreprint
Pawel, S., & Held, L. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-196009 · Cited by 2
Lehberger, M., & Grüner, S. (2020) · doi:10.31235/osf.io/5zc62 · Cited by 2
Closing a Mental Account: The Realization Effect for Gains and LossesPreprint
Merkle, C., MMller-Dethard, J., & Weber, M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3189076 · Cited by 2
Zeevi, Y., Astashenko, S., Mudrik, L., & Benjamini, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11585 · Cited by 4
Overprecision is a property of thinking systemsPreprint
Moore, D. A. (2019) · doi:10.31234/osf.io/fxswm · Cited by 2
When Linda meets Preeti: The Validation of Behavioral Biases in IndiaPreprint
Owsley, N. C. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/v2yjg · Cited by 3
What do Incoming University Students Believe About Open Science Practices in Psychology?
Beaudry, J. L., Williams, M. N., Philipp, M. C., & Kothe, E. J. (2022) · Teaching of Psychology · doi:10.31234/osf.io/rq3zs · Cited by 13
“Of course, psychology is not alone; the replication crisis extends to other fi elds, including economics (Camerer et al., 2016) and medicine (e.g., Errington et al., 2021).”
BackgroundNon-Standard ErrorsPreprint
Menkveld, A. J. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3979358 · Cited by 6
Fanelli, D., Tan, P. B., Amaral, O. B., et al. (2022) · doi:10.31222/osf.io/5r36g · Cited by 2
Thompson, J., Dreber, A., Gaunt, T. R., Gordon, M., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Lyon, M., Penton‐Voak, I. S., et al. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/gsc8f · Cited by 1
Self-citationCognitive Reflection and 2D:4D: Evidence from a Large Population SamplePreprint
Neyse, L., Fossen, F. M., Johanneson, M., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4063697 · Cited by 2
Self-citationResponsible Product Design to Mitigate Excessive Gambling: A Scoping ReviewPreprint
McAuliffe, W. H. B., Louderback, E. R., Edson, T. C., LaRaja, A. S., & LaPlante, D. A. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xf7ar · Cited by 1
Setting Up a Dofile for Team ResearchPreprint
McWay, R., Allen, J. E., & Yu, H. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3965114 · Cited by 2
Proposal to NSF 19-565 to Create a STEM Education Research HubPreprint
Nosek, B. A., Ofiesh, L., Grasty, F. L., Pfeiffer, N., Mellor, D. T., Brooks, R. E., Benjamin, M., Booth, A., Bowman, S. D., Davis, J. C., et al. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/4mpuc · Cited by 3
The impact of P-hacking on "Redefine statistical significance"Preprint
Crane, H. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bp2z4 · Cited by 2
Toward open and reproducible epidemiologyPreprint
Mathur, M. B., & Fox, M. P. (2022) · doi:10.31219/osf.io/bpkf7 · Cited by 2
Preregistration and increased transparency will benefit science
Mellor, D. T. (2017) · European Science Editing · doi:10.31219/osf.io/xsfam · Cited by 3
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v5 · Cited by 3
Thibault, R. T., Kovacs, M., Hardwicke, T. E., et al. (2023) · doi:10.31222/osf.io/md2xz · Cited by 2
Silverstein, P., Pennington, C. R., Branney, P., et al. (2023) · doi:10.31234/osf.io/b5m6q · Cited by 2
Poor hypotheses and research waste in biology: learning from a theory crisis in psychologyPreprint
Nakagawa, S., Armitage, D., Froese, T., et al. (2024) · doi:10.32942/x2s03w · Cited by 2
The Perks and Perils of Machine Learning in Business and Economic ResearchPreprint
Dudda, T. L., & Hornuf, L. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4981802 · Cited by 3
Huber, C., Holzmeister, F., Johannesson, M., König-Kersting, C., Dreber, A., Huber, J., & Kirchler, M. (2026) · The Journal of Finance · doi:10.2139/ssrn.5048949 · Cited by 5
Self-citationLeaning In Softly: On Gender and Self-PromotionPreprint
Chang, J., Saccardo, S., & Gallus, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5240652 · Cited by 2
A New Standard for the Analysis and Design of Replication Studies
Held, L. (2019) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12493 · Cited by 80
ЕПОХА ПОСТПРАВДИ У СУЧАСНОМУ СВІТІ: ВИТОКИ ТА ПРИЧИНИ
Коркос, Я. (2023) · Слобожанський науковий вісник. Серія Психологія. · doi:10.32782/psyspu/2023.1.4 · Cited by 1
Power and bias in industrial relations research
Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2023) · British Journal of Industrial Relations · doi:10.1111/bjir.12746 · Cited by 3
Hensel, P. G., & Kacprzak, A. (2024) · Journal of Organizational Change Management · doi:10.1108/jocm-10-2023-0436 · Cited by 1
Agrillo, C., Rovegno, E., Dadda, M., Bertolucci, C., & Bisazza, A. (2024) · Animals · doi:10.3390/ani14243684 · Cited by 1
“In recent years, the replicability of scientific findings and the reproducibility of experimental procedures have become critical topics in the scientific community [47,48].”
Religion, Brain & Behavior adopts stricter transparency standards
Wildman, W. J., Ross, R. M., McKay, R., Watts, J., Hoogeveen, S., Sosis, R. H., Cristofori, I., Price, M. E., & Shaver, J. H. (2024) · Religion Brain & Behavior · doi:10.1080/2153599x.2024.2424031 · Cited by 1
“In economics, Camerer et al. (2016) successfully replicated two thirds of 18 selected experimental studies from top journals ( Quarterly Journal of Economics and American Economic Review ).”
Do hedge funds still manipulate stock prices?
Cui, X., & Kolokolova, O. (2025) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2025.102765 · Cited by 2
Diversity and a taste for power
Meagher, K., & Wait, A. (2025) · The Journal of Law Economics and Organization · doi:10.1093/jleo/ewaf007 · Cited by 2
Messer, K. D., Ganguly, D., & Xie, L. (2025) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13519 · Cited by 2
Lee, G. K. (2025) · Strategic Management Review · doi:10.1561/111.00000081 · Cited by 4
Alexiou, C., & Saridakis, G. (2025) · Journal of Entrepreneurship and Innovation in Emerging Economies · doi:10.1177/23939575251335084 · Cited by 4
“For instance, it has been found that in experimental economics, the rate of replication in prestigious economic scientific journals (i.e., American Economic Review , Quarterly Journal of Economics ) is between 61% and 78% (e.g., Camerer et al., 2016).”
Reducing the causal illusion: a question of motivation or of information?
Vinas, A., Blanco, F., & Matute, H. (2025) · Royal Society Open Science · doi:10.1098/rsos.250082 · Cited by 1
“Nevertheless, we believe that our results provide additional evidence to the debate on whether financial incentives are indispensable in experimentation, both in Psychology and in Economics (see [23–25,42”
Caicedo, J., Guirado, C., González, M. C., & Walker, J. (2025) · Transport Policy · doi:10.1016/j.tranpol.2025.06.009 · Cited by 3
McWay, R., & Braaksma, M. (2025) · World Development Perspectives · doi:10.1016/j.wdp.2025.100707 · Cited by 1
Schweizer, G., & Köppel, M. (2025) · Europe’s Journal of Psychology · doi:10.5964/ejop.14957 · Cited by 1
“…replication projects, namely the Re producibility Project in Psychology (RPP; OSC, 2015), the Experimental Economics Replication Project (EERP; Camerer et al., 2016), Many Labs 1 (ML 1; Klein et al., 2014) and Many Labs 3 (ML 3; Ebersole et al., 2016), totaling 131 attempts at replicating…”
Williams, R. I., Manley, S. C., Aaron, J. R., & Daniel, F. (2025) · Journal of Small Business Strategy · doi:10.53703/001c.142443 · Cited by 2
Hypothetical nudges provide directional but noisy estimates of real behavior change
Gandhi, L., Kiyawat, A., Camerer, C. F., & Watts, D. J. (2025) · Communications Psychology · doi:10.1038/s44271-025-00339-x · Cited by 5
Self-citationLarge-scale community study reveals information sampling drives fairness decisions
Vahed, S., & Sanfey, A. (2025) · Communications Psychology · doi:10.1038/s44271-025-00354-y · Cited by 1
“Taken together, our methodology occupies an intermediate space between traditional, controlled experiment and large-scale observational study 32 , aiming to engage longstanding concerns regarding the ecological validity and generalisability of laboratory-based decision research 2 .”
Haim, M., Knöpfle, P., & Breuer, J. (2025) · Media Psychology · doi:10.1080/15213269.2025.2595452 · Cited by 1
A pre-registration template for transparent choice experiment research
Sagebiel, J., Schulze, C., & Rommel, J. (2026) · Q Open · doi:10.1093/qopen/qoag026 · Cited by 1
Do “Evidence‐Based Policy” Clearinghouses Provide Good Advice for Local Policymakers?
Orr, L. L. (2025) · Journal of Policy Analysis and Management · doi:10.1002/pam.70077 · Cited by 1
Ørbeck, M., Lindhjem, H., Kipperberg, G., & Loureiro, M. L. (2026) · Resource and Energy Economics · doi:10.1016/j.reseneeco.2026.101569 · Cited by 2
Bennett, H., Blake, H. T., d’Unienville, N. M. A., Murray, J., & Fox, J. L. (2026) · Sports Medicine · doi:10.1007/s40279-026-02451-2 · Cited by 2
“An inability to replicate published research has been demonstrated in the field of psychology [7–10], cancer [11], economics [12], and philosophy [13].”
Kondzic, D., Pawel, S., Hoffmann, J., Twardawski, M., Held, L., Sengewald, M.-A. D.-P., & Pohl, S. (2026) · Royal Society Open Science · doi:10.1098/rsos.252263 · Cited by 1
Open Science in den Wirtschaftswissenschaften: Transparenz, Reproduzierbarkeit und Zugang
Schmidt, K. M., Neyse, L., Saam, M., Siegfried, D., Vilhuber, L., & Winter, J. (2026) · Perspektiven der Wirtschaftspolitik · doi:10.1515/pwp-2026-0019 · Cited by 2
“…zwischen den Disziplinen: In der Psychologie waren die Replikationsraten besonders niedrig, während experimentelle Studien in den Wirtschaftswissenschaften im Durchschnitt etwas stabilere Ergebnisse liefern, wenngleich auch hier reduzierte Effekt-stärken beobachtet wurden (Camerer et al. 2016).”
What Lessons Does the ‘Replication Crisis’ in Psychology Hold for Experimental Economics?
Bardsley, N. (2018) · Cambridge University Press eBooks · doi:10.1017/9781316676349.003 · Cited by 2
Replikationen, Reputation und gute wissenschaftliche Praxis
Fecher, B., Fräßdorf, M., Hebing, M., et al. (2017) · Information - Wissenschaft & Praxis · doi:10.1515/iwp-2017-0025 · Cited by 1
Sibling correlation in risk attitudes: evidence from Burkina Faso
Sepahvand, M. H., & Shahbazian, R. (2020) · The Journal of Economic Inequality · doi:10.1007/s10888-020-09466-3 · Cited by 1
“This is important, since recently replicability and reproducibility of scientific findings have remerged as a salient factor (Dreber et al., 2015; Camerer et al., 2016), by being able to use and analyse the same measures as previous studies.”
BackgroundResultMaking our “meta-hypotheses” clear: heterogeneity and the role of direct replications in science
Strømland, E. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00348-7 · Cited by 1
On the Importance of Replicating Experiments in Economics
Migheli, M. (2021) · Social Epistemology · doi:10.1080/02691728.2021.1953187 · Cited by 1
“Castillo (2015), Camerer et al. (2016) and Camerer et al.”
BackgroundInfluentialFunkcjonowanie poznawcze w uzależnieniach behawioralnych
Nowakowska, I., Lewandowska, K., & Lewczuk, K. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial04 · Cited by 1
Key methodological issues in researching gifted education and advanced academics
McBee, M. T. (2021) · Routledge eBooks · doi:10.4324/9781003025443-1 · Cited by 1
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 3
Does competitive winning increase subsequent cheating?
Colman, A. M., Pulford, B. D., Frosch, C. A., Mangiarulo, M., & Miles, J. N. V. (2022) · Royal Society Open Science · doi:10.1098/rsos.202197 · Cited by 1
2020 Rossi Award Lecture: The Evolving Art of Program Evaluation
Brown, R. (2022) · Evaluation Review · doi:10.1177/0193841x221121241 · Cited by 1
“These variations in how interventions are delivered often lead to nonreplicability of favorable effects found in an initial evaluation of a program (Camerer et al., 2016; Ioannidis et al., 2017; Mahoney, 2010).”
BackgroundFostering innovation: Experimental evidence on the effectiveness of behavioral interventions
Matthewes, E., Nassar, A., & Zihlmann, C. (2022) · PLoS ONE · doi:10.1371/journal.pone.0276463 · Cited by 3
“Also, replicated studies are documented to have on average a 33% lower effect size than the original studies [67].”
How often are replication attempts questioned?
Hensel, P. G. (2023) · Accountability in Research · doi:10.1080/08989621.2023.2198126 · Cited by 1
“…replicability of research in the field of psychology (O’donnell et al. 2021; Open Science Collaboration 2015) and other social sciences (Camerer et al. 2016, 2018) has resulted in a new impetus for supporting replication research (Hales, Wesselmann, and Hilgard 2019; Zwaan et al. 2018).”
Akamatsu, D. (2022) · Kodo Keiryogaku (The Japanese Journal of Behaviormetrics) · doi:10.2333/jbhmk.49.113 · Cited by 1
The replication crisis in science and its overlap into kinantropology: an introduction
Baloun, L., & Mališ, J. (2023) · Tělesná kultura · doi:10.5507/tk.2023.002 · Cited by 1
Gruen, A., Mattingly, K., Morwitch, E., Bossaerts, F., Clifford, M., Nash, C., Ioannidis, J. P. A., & Ponsonby, A. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104783 · Cited by 4
“The Next Generation Social Science program (NGS2) 15 employed this system successfully, pooling 103 markets from four large-scale scientific replication projects across psychology 10 , economics 16 , and social science 17 .”
Integrating measures of replicability into scholarly search: Challenges and opportunities
Wu, C., Chakravorti, T., Carroll, J. M., & Rajtmajer, S. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3643043 · Cited by 7
“These have included efforts targeting psychology [35], economics [13, 24], experimental philosophy [39], social and behavioral sciences more broadly [25], cancer biology [50], neuroscience [18], computer science [36], and machine learning [99].”
Intergenerational Transformation of Empirical Research in Economics
Sawada, Y. (2024) · Springer eBooks · doi:10.1007/978-981-97-1887-0_2 · Cited by 1
Investigation of the convex time budget experiment by parameter recovery simulation
Inukai, K., Shimodaira, Y., & Shiozawa, K. (2024) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2024.100962 · Cited by 1
Is generalisation hindering the adoption of your findings?
Heldal, R. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686694 · Cited by 1
“These methods usually come with their own challenges, such as replicability problems [6, 10, 22, 36].”
Chatziathanasiou, K. (2024) · Springer eBooks · doi:10.1007/978-3-662-69010-9_10 · Cited by 1
Expectations and social decision making: An investigation of gain and loss ultimatum games
Heilman, R. M., Ursu, A., Trif, S., Kusev, P., Martin, R., & Teal, J. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102313 · Cited by 3
Re-examining the effect of wage delegation: a replication study of Charness et al. (2012)
Niehoff, H., & Schreck, P. (2024) · Journal of Business Economics · doi:10.1007/s11573-024-01214-7 · Cited by 1
“For instance, the findings of a large-scale replication project conducted by Camerer et al. (2016) demonstrated that only 11 out of 18 replicated studies (61”
Do replications receive fewer citations? A counterfactual approach
Coupé, T., Logchies, T., & Reed, W. R. (2025) · Scientometrics · doi:10.1007/s11192-025-05288-0 · Cited by 3
“There is widespread concern in many fields of science that there exists a “reproducibility crisis” (Begley & Ellis, 2012; Camerer et al., 2016, 2018; Duvendack et al., 2015; Err-ington et al., 2021; Ioannidis, 2005a, 2005b; Open Science Collaboration, 2015; Prinz et al., 2011; Van Noorden, 2023).”
ChatGPT struggles to recognize reproducible science
Chang, J. R., & Nordling, T. E. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02428-z · Cited by 3
Drouvelis, M., & Qiu, Z. (2025) · Economic Inquiry · doi:10.1111/ecin.13297 · Cited by 4
The Prevalence of Replication Studies in Human Factors (2020-2024)
Doane, R., Andrade, F. M. G., Marra-Dotts, J., Li, B., Groetsch, S., Lichter, K. A., Do, A., Hyk, A., Baker, S., Johnson, J. D., et al. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251376984 · Cited by 1
“Alarmingly, large-scale replication efforts have revealed strikingly low replication rates in areas of social/behavioral science (e.g., Camerer et al., 2016; Klein et al., 2018; Open Science Collaboration, 2015), neuroscience (e.g., Boekel et al., 2015), and oncology (e.g., Errington et al., 2021),…”
Spreckelsen, T. F., & van der Horst, M. (2016) · Sociological Research Online · doi:10.5153/sro.4076 · Cited by 2
“, sometimes study and analysis preregistration is required, and we can see a start of a replication culture (Bohannon 2015; Camerer et al. 2016).”
BackgroundLec, F. L., & Macé, S. (2017) · Theory and Decision · doi:10.1007/s11238-017-9621-0 · Cited by 1
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.