Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 201–300 of 1,372
Heterogeneity in banker culture and its influence on dishonesty
Rahwan, Z., Yoeli, E., & Fasolo, B. (2019) · Nature · doi:10.1038/s41586-019-1741-y · Cited by 42
Reproducibility and Replicability in Economics
Vilhuber, L. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4f6b9e67 · Cited by 71
Escaping Paternalism: Rationality, Behavioral Economics, and Public Policy
Rizzo, M. J., & Whitman, G. (2019) · Cited by 42
A standard protocol for describing the evaluation of ecological models
Planque, B., Aarflot, J. M., Buttay, L., Carroll, J., Fransner, F., Hansen, C., Husson, B., Langangen, Ø., Lindstrøm, U., Pedersen, T., et al. (2022) · Ecological Modelling · doi:10.1016/j.ecolmodel.2022.110059 · Cited by 48
“The crisis has primarily been identified in the fields of psychology (Pashler and Wagenmakers 2012), clinical studies (Begley and Ioannidis John 2015) and economics (Camerer et al., 2016) and is much less discussed in ecological research (but see Ives 2018; Nichols et al., 2019, 2021).”
BackgroundThe future of academic publishing
Ahmed, A., Al-Khatib, A., Boum, Y., Debat, H., Gurmendi Dunkelberg, A., Hinchliffe, L. J., Jarrad, F., Mastroianni, A., Mineault, P., Pennington, C. R., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01637-2 · Cited by 42
Stracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 54
Replications in agricultural economics
Finger, R., Grebitus, C., & Henningsen, A. (2023) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13386 · Cited by 56
Numerical data quality in IS research and the implications for replication
Marsden, J. R., & Pingry, D. E. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.10.007 · Cited by 43
Tsui, A. S. (2021) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-062021-021303 · Cited by 49
Foundations and Futures of Strategic Management
Leiblein, M. J., & Reuer, J. J. (2020) · Strategic Management Review · doi:10.1561/111.00000001 · Cited by 73
Towards more evidence-based agricultural and food policies
Benni, N. E., Grovermann, C., & Finger, R. (2023) · Q Open · doi:10.1093/qopen/qoad003 · Cited by 59
“For example, Camerer et al. ( 2016 ) showed that 40 per cent of economic experiments published in top economic journals failed to be replicable.”
BackgroundCueva, C., Roberts, E., Spencer, T. J., Rani, N., Tempest, M., Tobler, P. N., Herbert, J., & Rustichini, A. (2016) · Hormones and Behavior · doi:10.1016/j.yhbeh.2016.09.012 · Cited by 42
“This issue has been highlighted by recent large replication attempts in psychology and experimental economics (Open Science Collaboration, 2015; Camerer et al., 2016) aswell as by recent critical reviews of oxytocin research on humans (Nave et al., 2015; Lane et al., 2016).”
BackgroundMethodHonest signaling in academic publishing
Tiokhin, L., Panchanathan, K., Lakens, D., Vazire, S., Morgan, T. J. H., & Zollman, K. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246675 · Cited by 46
“For example, publications in high-ranking journals are preferentially chosen for direct replication [21–23] and may be more heavily scrutinized for errors and fraud [24, 37], which increases the probability that low-quality papers in high-ranking journals are detected.”
Ebert, D. D., Franke, M., Zarski, A., Berking, M., Riper, H. M., Cuijpers, P. J., Funk, B., & Lehr, D. (2021) · Journal of Medical Internet Research · doi:10.2196/22107 · Cited by 40
Hantula, D. A. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00194-2 · Cited by 45
Primer for Experimental Methods in Organization Theory
Levine, S. S., Schilke, O. S., Kacperczyk, A. J., & Zucker, L. G. (2023) · Organization Science · doi:10.1287/orsc.2023.18093 · Cited by 42
“…of findings in the social sciences at large (Camerer et al. 2018), psychology (e.g., Open Science Collaboration 2015, Ebersole et al. 2016, Klein et al. 2018), economics (Camerer et al. 2016), and most recently, landmark experiments published in Management Science (Davis et al. 2023).”
Physics of human cooperation: experimental evidence and theoretical models
Sánchez, Á. (2018) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/aaa388 · Cited by 47
Multiple Facets of Value-Based Decision Making in Major Depressive Disorder
Mukherjee, D., Lee, S., Kazinka, R., Satterthwaite, T. D., & Kable, J. W. (2020) · Scientific Reports · doi:10.1038/s41598-020-60230-z · Cited by 49
Evidence of general economic principles of bargaining and trade from 2,000 classroom experiments
Lin, P.-H., Brown, A. L., Imai, T., Wang, J. T., Wang, S. W., & Camerer, C. F. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0916-8 · Cited by 50
Self-citationHow effective are social engineering interventions? A meta-analysis
Bullée, J.-W., & Junger, M. (2020) · Information and Computer Security · doi:10.1108/ics-07-2019-0078 · Cited by 60
Block, J., Fisch, C. O., Kanwal, N., Lorenzen, S., & Schulze, A. (2022) · Management Review Quarterly · doi:10.1007/s11301-022-00269-6 · Cited by 46
Identification of and correction for publication biasPreprint
Kasy, M., & Andrews, I. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/49yst · Cited by 42
Toward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 56
“At that time, efforts to independently replicate published studies—that is, to collect and analyze new data using a similar protocol—often yielded results that did not support original studies’ conclusions (2, 3).”
ResultNakagawa, S., Lagisz, M., Yang, Y., & Drobniak, S. M. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002423 · Cited by 43
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 67
Self-citationInvestigating the replicability of the social and behavioural sciences
Tyner, A. H., Abatayo, A. L., Daley, M., Field, S., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Miske, O., et al. (2026) · Nature · doi:10.1038/s41586-025-10078-y · Cited by 60
“These findings are consistent with the cumulative evidence across systematic replications in the social and behavioural sciences 3,5,10 and from other fields 13 , and they illustrate that there is substantial uncertainty in estimating replicability.”
Uninformative Anchors Have Persistent Effects on Valuation Judgments
Yoon, S., & Fong, N. M. (2019) · Journal of Consumer Psychology · doi:10.1002/jcpy.1091 · Cited by 42
The entitlement effect in the ultimatum game – does it even exist?
Demiral, E. E., & Möllerström, J. (2018) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2018.08.022 · Cited by 43
Herz, H., & Taubinsky, D. (2016) · Journal of the European Economic Association · doi:10.1093/jeea/jvx011 · Cited by 50
“In the context of risk preferences, Ericson et al. (2011); Heffetz and List (2014) and Camerer et al. (2016) provide (mixed) evidence that expectations shape reference points, which is also demonstrated by Abeler et al. (2011) and Camerer et al. (2016) in the context of effort provision.”
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 41
“…the original analysis with the original data; Chang et al., 2015; Hardwicke et al., 2018; Ioannidis et al., 2009) or replicated (verified through conducting the original analysis with newly collected data; Camerer et al., 2016, 2018; Cova et al., 2021; Open Science Collaboration [OSC], 2015).”
MethodThe Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 45
“…infrequently.1 Recently, a number of systematic, large-scale replication attempts have been published in prominent scientific journals (e.g., see Camerer et al., 2016, 2018; Klein et al., 2018; Open Science Collaboration, 2015), and the question of replicability has received a substantial…”
BackgroundThe robustness of anchoring effects on preferential judgments
Yoon, S., Fong, N. M., & Dimoka, A. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500006148 · Cited by 41
Assessing heterogeneity and power in replications of psychological experiments.
Schauer, J. M., & Hedges, L. V. (2020) · Psychological Bulletin · doi:10.1037/bul0000232 · Cited by 38
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
The Significance of Data-Sharing Policy
Askarov, Z., Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2022) · Journal of the European Economic Association · doi:10.1093/jeea/jvac053 · Cited by 49
A Replication Study of Operations Management Experiments in Management Science
Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S. M., Leider, S., Long, X., & Tong, J. D. (2022) · Management Science · doi:10.1287/mnsc.2023.4866 · Cited by 54
Number Effects and Tacit Collusion in Experimental Oligopolies
Horstmann, N., Krämer, J., & Schnurr, D. (2018) · Journal of Industrial Economics · doi:10.1111/joie.12181 · Cited by 32
A partial solution for the replication crisis in economics
Briggs, W. M. (2023) · Asian Journal of Economics and Banking · doi:10.1108/ajeb-03-2023-0027 · Cited by 39
Cui, Z., Li, N., & Zhou, H. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00840-7 · Cited by 121
“We standardized the direction of effects and converted various reported effect size metrics—such as Cohen’s d , eta-square, F statistics, and chi-square—into correlation coefficients ( r ) for consistency and to facilitate interpretation 26 .”
Design and Analysis of Replication Studies
Bonett, D. G. (2020) · Organizational Research Methods · doi:10.1177/1094428120911088 · Cited by 39
“In a replication study of 18 economic studies (Camerer et al., 2016), about 39% of the follow-up studies had inconclusive directional nonreplication but were interpreted as replication failures. In a replication study of 10 psychology experiments (Ebersole et al., 2016), 70% of the studies were described as replicatio…”
BackgroundReplications in Development Economics
Sukhtankar, S. (2017) · American Economic Review · doi:10.1257/aer.p20171120 · Cited by 35
“In economics, Camerer et al. (2016) attempted to replicate 18 papers in experimental economics. To aid efforts in replication, all of the “Top 5” journals in economics have now adopted a Data Access Policy (DAP), requiring authors to make available data and code to all researchers for the purposes of replication.(1) Y…”
BackgroundThe role of replication in psychological science
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00329-2 · Cited by 49
Radical Solutions and Open Science: An Open Approach to Boost Higher Education
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 51
Advances in transparency and reproducibility in the social sciences
Freese, J., Rauf, T., & Voelkel, J. G. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102770 · Cited by 60
Predicting replication outcomes in the Many Labs 2 studyPreprint
Forsell, E., Viganola, D., Pfeiffer, T., et al. (2018) · doi:10.31234/osf.io/sgjaz · Cited by 34
Self-citationChopra, F., Haaland, I., Roth, C., & Stegmann, A. (2023) · The Economic Journal · doi:10.1093/ej/uead060 · Cited by 85
Probabilistic forecasting of replication studies
Pawel, S., & Held, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231416 · Cited by 46
“There have been many attempts to forecast the outcomes of replication studies based on the results from the original studies [5, 7, 12–15].”
InfluentialLegal access to alcohol and criminality
Hansen, B., & Waddell, G. R. (2017) · Journal of Health Economics · doi:10.1016/j.jhealeco.2017.08.001 · Cited by 37
“Indeed, Camerer et al. (2016) found only 11 of 18 expermental studies published in The American Economic Review and The Quarterly Journal f Economics could be reproduced, while Chang and Li (2015) found only 29 of 59 f econometric studies that that used observational data could be replicated, even…”
Bayesian markets to elicit private information
Baillon, A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1703486114 · Cited by 38
“Prediction markets have been used by various public organizations and companies (1, 2); for example, social scientists use this type of market to predict the replicability of experiments (3, 4).”
BackgroundTesting the Waters: Behavior across Participant Pools
Snowberg, E., & Yariv, L. (2018) · National Bureau of Economic Research · doi:10.3386/w24781 · Cited by 37
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32
Sunlit Uplands: The Genius of the NICE Reference Case
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.435 · Cited by 34
Wuttke, A. (2018) · Politische Vierteljahresschrift · doi:10.1007/s11615-018-0131-7 · Cited by 46
fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
Yücel, M. A., Luke, R., Mesquita, R. C., von Lühmann, A., Mehler, D. M. A., Lührs, M., Gemignani, J., Abdalmalak, A., Albrecht, F., Ivo, I. D. A., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08412-1 · Cited by 57
DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments
Alberti, M., Pondenkandath, V., Würsch, M., Ingold, R., & Liwicki, M. (2018) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr-2018.2018.00080 · Cited by 36
“The replication crisis has been widely studied in other fields [1], [2], [3] and is getting more attention in the machine learning community, recently [4], [5].”
BackgroundData Science Support at the Academic Library
Oliver, J. C., Kollen, C., Hickson, B., & Rios, F. (2019) · Journal of Library Administration · doi:10.1080/01930826.2019.1583015 · Cited by 33
“There have been several studies in a wide variety of disciplines that have demonstrated how a large number of results in published studies cannot be reproduced (Aarts et al., 2015; Camerer et al., 2016; Ioannidis, 2005).”
BackgroundEgo Depletion May Disappear by 2020
Vadillo, M. A. (2019) · Social Psychology · doi:10.1027/1864-9335/a000375 · Cited by 33
The Impact of Complexity on Methods and Findings in Psychological Science
Sanbonmatsu, D. M., Cooley, E. H., & Butner, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580111 · Cited by 50
Rodgers, P., & Collings, A. (2021) · eLife · doi:10.7554/elife.75830 · Cited by 34
Towards Bayesian Model-Based Demography: Agency, Complexity and Uncertainty in Migration Studies
Bijak, J. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7 · Cited by 47
Mullen, P. D., Savas, L. S., Bundy, Ł., Haardörfer, R., Hovell, M., Fernández, M. E., Monroy, J., Williams, R. S., Kreuter, M. W., Jobe, D., et al. (2016) · Tobacco Control · doi:10.1136/tobaccocontrol-2016-053045 · Cited by 34
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 48
New Statistical Metrics for Multisite Replication Projects
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12572 · Cited by 28
“…2014) has proposed analytic approaches for a single replication of a single study or designs in which numerous original studies across a discipline or domain are each replicated once (here termed ‘one-to-one designs’), as in Open Science Collaboration (2015) and in Camerer et al. (2016).”
Background‘I’m Just a Soul Whose Intentions Are Good’: The Role of Communication in Noisy Repeated Games
Arechar, A. A., Dreber, A., Drew Fudenberg, B., & Rand, D. G. (2016) · Games and Economic Behavior · doi:10.1016/j.geb.2017.06.013 · Cited by 42
“Given that the experimental setup is identical between the two papers, it seems likely that this difference reflects differences in participant pool (Nottingham vs Harvard) or in the sequence of game lengths implemented, particularly given Camerer et al. (2016)’s nearly exact replication of the Fudenberg et al. (2012)…”
InfluentialSelf-citationThe assessment of replication success based on relative effect size
Held, L., Micheloud, C., & Pawel, S. (2022) · The Annals of Applied Statistics · doi:10.1214/21-aoas1502 · Cited by 41
“The second data set comes from the Experimental Economics Replication Project (Camerer et al., 2016) which attempted to replicate 18 experimental economics studies published in two high impact economics journals between 2011 and 2015.”
BackgroundAre Reference Points Merely Lagged Beliefs Over Probabilities?
Heffetz, O. (2018) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2020.11.010 · Cited by 32
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
Modeling Imaginary Worlds: Version 4 of the AMCP Format for Formulary Submissions
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.434 · Cited by 27
Risk Aversion and Son Preference: Experimental Evidence from Chinese Twin Parents
Chew, S. H., Yi, J., Zhang, J., & Zhong, S. (2017) · Management Science · doi:10.1287/mnsc.2017.2779 · Cited by 36
“Camerer et al. (2016) replicated 18 studies published in in the American Economic Review and the Quarterly Journal of Economics in 2011-2014, and show a replication rate of about 67 percent to 78 percent. Relatedly, Maniadis, Tufano, and List (2014) demonstrate that the probability of false positives decreases with th…”
BackgroundInfluentialReproducible research: a minority opinion
Drummond, C. (2017) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2017.1413140 · Cited by 39
Use of reproducible research practices in public health: A survey of public health analysts
Harris, J. K., Johnson, K. J., Carothers, B. J., Combs, T., Luke, D. A., & Wang, X. (2018) · PLoS ONE · doi:10.1371/journal.pone.0202447 · Cited by 40
P in the right place: Revisiting the evidential value of P‐values
Lytsy, P. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12319 · Cited by 39
A Future for Observational Epidemiology: Clarity, Credibility, Transparency
Harper, S. B. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy280 · Cited by 33
The transformative service paradox: the dilemma of wellbeing trade-offs
Russell‐Bennett, R., Mulcahy, R. F., Letheren, K., McAndrew, R., & Dulleck, U. (2020) · Journal of service management · doi:10.1108/josm-10-2019-0324 · Cited by 41
On the stress potential of an organisational climate of innovation: a survey study in Germany
Fischer, T., & Riedl, R. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1836258 · Cited by 51
“Moreover, replication of existing research fi ndings is a signi fi cantly neglected topic in research in general (Camerer et al. 2016, 2018) and speci fi cally in IS research (Dennis and Valacich 2014).”
BackgroundHeckelei, T., Hüttel, S., Rommel, J., & Odening, M. (2023) · German Journal of Agricultural Economics · doi:10.30430/gjae.2023.0231 · Cited by 49
“C AMERER et al., 2016).”
Extending the Cooperative Phenotype: Assessing the Stability of Cooperation across Countries
Reigstad, A. G., Strømland, E., & Tinghög, G. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01990 · Cited by 30
“Recent replication failures in the behavioral sciences (Open Science Collaboration, 2015; Camerer et al., 2016) underscore the importance of replicating previous work.”
BackgroundPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Evaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M. (2017) · doi:10.31219/osf.io/5y8w7 · Cited by 29
Mayrhofer, R., Büchner, I. C., & Hevesi, J. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1390233 · Cited by 34
“In other disciplines such as medicine (e.g., Ioannidis, 2005), economics (e.g., Camerer et al., 2016), natural sciences and engineering (e.g., Baker, 2016), it has also been found that only some of the published results can be replicated.”
BackgroundIntroduction: Forum on Reproducibility and Replicability in Geography
Goodchild, M. F., Fotheringham, A. S., Kedron, P., & Li, W. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806030 · Cited by 30
The Sceptical Bayes Factor for the Assessment of Replication Success
Pawel, S., & Held, L. (2022) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12491 · Cited by 32
Predicting and reasoning about replicability using structured groupsPreprint
Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Wilkinson, D. P., Hemming, V., Bush, M., Fraser, H. S., Thorn, F. S., McBride, M. F., et al. (2021) · doi:10.31222/osf.io/vtpmb · Cited by 28
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28
“Fields like psychology [13], biomedical research [14,15], economics [16] and broader social sciences [17] all witnessed ‘many-lab’ studies whose findings indicated levels of reproducibility ranging between 30 and 70%.”
A Skeptical View of the National Science Foundation's Role in Economic Research
Cowen, T., & Tabarrok, A. T. (2016) · The Journal of Economic Perspectives · doi:10.1257/jep.30.3.235 · Cited by 28
Randomizing Development: Method or Madness?
Pritchett, L. H. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0004 · Cited by 53
Pricey therefore good? Price affects expectations, but not quality perceptions and liking
Kurz, J., Efendić, E., & Goukens, C. (2023) · Psychology and Marketing · doi:10.1002/mar.21799 · Cited by 40
Cooperation in the Prisoner’s Dilemma: an experimental comparison between pure and mixed strategies
Heuer, L., & Orland, A. (2019) · Royal Society Open Science · doi:10.1098/rsos.182142 · Cited by 36
Best practices in replication: a case study of common information in coordination games
Chen, R., Chen, Y., & Riyanto, Y. E. (2020) · Experimental Economics · doi:10.1007/s10683-020-09658-8 · Cited by 28
“In at least three out of the seven “failed” replications in Camerer et al. (2016), the replication subject pool might not have been culturally appropriate.”
BackgroundMethodResultInfluentialMalmendier, U. M. (2018) · National Bureau of Economic Research · doi:10.3386/w25162 · Cited by 83
Small effects: The indispensable foundation for a cumulative psychological sciencePreprint
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2021) · doi:10.31234/osf.io/hzrxf · Cited by 28
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
“So far, there does not seem to be a consensus on a single golden standard to quantify replication success, which explains why typically several metrics are reported [12,13,15,18].”
BackgroundMethodInfluentialShould We Strive to Make Science Bias-Free? A Philosophical Assessment of the Reproducibility Crisis
Hudson, R. G. S. (2021) · Journal for General Philosophy of Science · doi:10.1007/s10838-020-09548-w · Cited by 33
Wang, C. S., Whitson, J., King, B. G., & Ramirez, R. L. (2021) · Organization Science · doi:10.1287/orsc.2021.1492 · Cited by 35
Preferences and perceptions in Provision and Maintenance public goods
Gächter, S., Kölle, F., & Quercia, S. (2022) · Games and Economic Behavior · doi:10.1016/j.geb.2022.06.009 · Cited by 39
Experimental Economics: Past and Future
Fréchette, G. R., Sarnoff, K., & Yariv, L. (2022) · Annual Review of Economics · doi:10.1146/annurev-economics-081621-124424 · Cited by 33
The effect of cognitive load on economic decision-making: a replication attempt
Ball, S. B., Katz, B., Li, F., & Smith, A. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.03.018 · Cited by 29
Managing and defending the commons: Experimental evidence from TURFs in Chile
Chávez, C. A., Murphy, J. J., & Stranlund, J. K. (2018) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2018.07.004 · Cited by 32
Cerulli-Harms, A., Goette, L. F., & Sprenger, C. D. (2019) · American Economic Journal Microeconomics · doi:10.1257/mic.20170271 · Cited by 47
Dawid, H., Harting, P., van der Hoog, S., et al. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2773539 · Cited by 27
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.