Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 801–900 of 1,372
Dunn, J., Mishra, V., Shandhi, M. M. H., Jeong, H., Yamane, N., Watanabe, Y., Chen, B. B., & Goodwin, M. S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1467424 · Cited by 5
Improving Replication in Endometrial Omics: Understanding the Influence of the Menstrual Cycle
Chung, J., & Rogers, P. A. W. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26020857 · Cited by 4
“This situation is similar to other fields (e.g., psychology [16], cancer biology [17], economics [18]), with each having replication crises where effect sizes tend to be dramatically diminished in replication studies or fail to replicate entirely.”
Fanelli, D., Tan, P. B., Amaral, O. B., & Neves, K. (2025) · Royal Society Open Science · doi:10.1098/rsos.241446 · Cited by 7
How to improve your relationship with your future self
Bowers, J., & Voors, M. J. (2016) · Revista de ciencia política · doi:10.4067/s0718-090x2016000300011 · Cited by 3
Yan-di, B. A. O., Ming, Z., Panchal, J. H., et al. (2020) · IEEE Access · doi:10.1109/access.2020.2971738 · Cited by 3
“It is important to replicate each other’s work for experimental researchers, since trust in scientific knowledge comes from the ability to independently replicate empirical findings [14].”
BackgroundReproducibility literature analysis - a federal information professional perspective
Antognoli, E., Avila, R. L., Sears, J., Christiansen, L. L., Tieman, J., & Hart, J. (2020) · IASSIST Quarterly · doi:10.29173/iq967 · Cited by 2
“Replication studies also help identify potential biases in the original study and serve as a basis for confirming or disconfirming prior findings (Spector, Johnson and Young, 2014) (Camerer et al., 2016).”
ResultGarchitorena, A., Murray, M. B., Hedt‐Gauthier, B., Farmer, P. E., & Bonds, M. H. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0007 · Cited by 5
Reproducibility in speech rate convergence experiments
Fuscone, S., Favre, B., & Prévot, L. (2021) · Language Resources and Evaluation · doi:10.1007/s10579-021-09528-6 · Cited by 5
Bias and Careers: Evidence from the Aid Effectiveness Literature
Doucouliagos, H., Hinz, T., & Zigová, K. (2020) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2021.102056 · Cited by 5
Thakur, P., & Jha, V. (2021) · Brazilian Journal of Otorhinolaryngology · doi:10.1016/j.bjorl.2021.11.004 · Cited by 4
Quantitative Macroeconomics: Lessons Learned from Fourteen Replications
Kirkby, R. (2022) · Computational Economics · doi:10.1007/s10614-022-10234-w · Cited by 5
“This current approach to replication in Quantitative Economics with it’s focus on reproducibility obviously misses any issues of whether the original results were themselves correct, which is the main purpose of replication.”
BackgroundInfluentialChin, J. M., Zeiler, K., Dilevski, N., Holcombe, A. O., Gatfield-Jeffries, R., Bishop, R., Vazire, S., & Schiavone, S. R. (2023) · F1000Research · doi:10.12688/f1000research.127563.1 · Cited by 8
Buckley, J., Araujo, J. A., Aribilola, I., Arshad, I., Azeem, M., Buckley, C., Fagan, A., Fitzpatrick, D. P., Herrera, D. A. G., Hyland, T., et al. (2023) · International Journal of Technology and Design Education · doi:10.1007/s10798-023-09827-9 · Cited by 4
“…fields such as psychology (Open Science Collaboration, 2015), cancer biology (Begley & Ellis, 2012; Errington et al., 2021), experimental economics (Camerer et al., 2016), and experimental philosophy (Cova et al., 2021), and refers to the identification of large percentages of original findings…”
On the generalizability of using mobile devices to conduct economic experiments
Guo, Y., Shachat, J. M., Walker, M. J., & Wei, L. (2023) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2023.102057 · Cited by 3
Hüffmeier, J., & Mertes, M. (2023) · Meta-Psychology · doi:10.15626/mp.2022.3271 · Cited by 5
Social sciences in crisis: on the proposed elimination of the discussion section
Schoenegger, P., & Pils, R. (2023) · Synthese · doi:10.1007/s11229-023-04267-3 · Cited by 6
Gender differences in multiple-choice questions and the risk of losing points
Riukula, K. (2023) · Journal of the Finnish Economic Association · doi:10.33358/jfea.122157 · Cited by 6
“The replicability of some scientific findings has also recently been called into question (see, e.g. Camerer et al., 2016).”
BackgroundUntrustworthy Evidence in Dishonesty Research
Bartos̆, F. (2024) · Meta-Psychology · doi:10.15626/mp.2023.3987 · Cited by 5
Adena, M., & Huck, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303133 · Cited by 4
“Replications of once-estab-lished findings can greatly raise confidence in their veracity [49, 50] and boost poststudy probabilities [48].”
BackgroundGrebitus, C., & Hu, W. (2025) · Journal of the Agricultural and Applied Economics Association · doi:10.1002/jaa2.70011 · Cited by 3
Testing the effect of cooperative/competitive priming on the Prisoner’s Dilemma. A replication study
Belaus, A., Reyna, C., & Freidín, E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0209263 · Cited by 5
“’s replication [63] of 18 experiments in experimental economics also present such cases.”
BackgroundClassroom experiments as a replication device
So, T. (2020) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2020.101525 · Cited by 3
Assessing Scientific Research Papers with Knowledge Graphs
Sun, K., Qiu, Z., Salinas, A., Huang, Y., Lee, D.-H., Benjamin, D., Morstatter, F., Ren, X., Lerman, K., & Pujara, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531879 · Cited by 7
“Some other researchers, from a range of disciplines such as psychology [24], biomedicine [13], economics [11] and social sciences [3], revisited a variety of published scientific papers, manually assessed the credibility of them by conducting direct replication studies, and further confirmed the…”
BackgroundDoes response time predict withdrawal decisions? Lessons from a bank-run experiment
Kiss, H. J., Rodríguez-Lara, I., & Rosa-García, A. (2019) · Review of Behavioral Finance · doi:10.1108/rbf-07-2018-0070 · Cited by 3
A Statistical Model to Investigate the Reproducibility Rate Based on Replication Experiments
Pauli, F. (2018) · International Statistical Review · doi:10.1111/insr.12273 · Cited by 3
Transparency in empirical economic research
Pérez, C. B., & Brodeur, A. (2019) · IZA World of Labor · doi:10.15185/izawol.467 · Cited by 6
Les pratiques de recherche ouvertes en psychologie
Lantian, A. (2020) · Psychologie Française · doi:10.1016/j.psfr.2020.09.001 · Cited by 6
Bayesian Data Analysis for Software Engineering
Torkar, R., Furia, C. A., & FELDT, R. H. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion52605.2021.00140 · Cited by 3
A note on salience of own preferences and the consensus effect
Dohmen, T., Quercia, S., & Willrodt, J. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.02.017 · Cited by 3
“Typically, for replication exercises the effect size found in previous studies is used (see, e.g., OpenScienceCollaboration, ; Camerer et al., , ).”
Arvan, M., Bright, L. K., & Heesen, R. (2022) · The British Journal for the Philosophy of Science · doi:10.1086/719117 · Cited by 29
Studying the empirical implications of the liberalization of airport markets
Button, K. J. (2020) · Competition and Regulation in Network Industries · doi:10.1177/1783591720935767 · Cited by 3
“In particular fields, the issue has been raised by Camerer et al. (2016) in experimental economics, Hamermesh (2017) in labor economics and Sukhtankar (2017) in development economics.”
BackgroundA gene–brain–behavior basis for familiarity bias in source preference
Chark, R., Zhong, S., Tsang, S. Y., Khor, C. C., Ebstein, R. P., Xue, H., & Chew, S. H. (2022) · Theory and Decision · doi:10.1007/s11238-022-09871-2 · Cited by 4
“The overall laboratory experiments in economics have an excellent track record for robustness (Camerer et al., 2016) albeit some concerns have been, nevertheless, raised about replicability (Schultze et al., 2019).”
BackgroundAbove a Swamp: A Theory of High-Quality Scientific Production
Kiri, B., Lacetera, N., & Zirulia, L. (2015) · National Bureau of Economic Research · doi:10.3386/w21143 · Cited by 6
Information Sharing, Advice Provision or Delegation: What Leads to Higher Trust and Trustworthiness?Preprint
Özer, Ö., Subramanian, U., & Wang, Y. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2565577 · Cited by 3
“would otherwise be possible, and lead to results that are relatively more replicable, being at least as robust as other kinds of empirical economics (Camerer et al., 2016).”
Social Status and Economic Behavior
Ordyan, G. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.119884 · Cited by 3
Administrative Data Linking and Statistical Power Problems in Randomized Experiments
Tahamont, S., Jelveh, Z., Chalfin, A. J., Yan, S., & Hansen, B. (2019) · National Bureau of Economic Research · doi:10.3386/w25657 · Cited by 4
Interdisciplinary Open Science: What are the implications for educational technology research?Preprint
Alhadad, S. S. J., Searston, R. A., & Lodge, J. M. (2019) · doi:10.35542/osf.io/w2bg9 · Cited by 3
The Economics of Computational ReproducibilityPreprint
Colliard, J.-É., Hurlin, C., & Hurlin, C. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3418896 · Cited by 10
Relative Wealth Placement and Risk-Taking BehaviorPreprint
Hillebrandt, M.-A., & Steinorth, P. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3440690 · Cited by 4
A Journal-Based Replication of “Being Chosen to Lead”
Drazen, A., Almenberg, A. D., Ozbay, E. Y., & Snowberg, E. (2019) · National Bureau of Economic Research · doi:10.3386/w26444 · Cited by 4
“…replication attempts are published.6 In recent years, mass replications have been carried out by teams of scientists (for example, Open Science Collaboration, 2015; Camerer et al., 2016; Ebersole et al., 2016; Schweinsberg et al., 2016; Camerer et al., 2018; Cova et al., 2018; Klein et al., 2018).”
InfluentialPower calculations for replication studiesPreprint
Micheloud, C., & Held, L. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-205155 · Cited by 3
Contributions towards understanding and building sustainable sciencePreprint
Hartgerink, C. H. J. (2019) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/4wtpc · Cited by 3
“Besides in psychology, reproducibility problems have also been indicated in economics (Camerer et al. 2016) and medicine (Begley and Ellis 2012).”
BackgroundIntuitive Prosociality: Heterogeneous Treatment Effects or False Positive?Preprint
Strømland, E., & Torsvik, G. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/hrx2y · Cited by 3
The low statistical power of psychological research: Causes, consequences and potential remedies
Thorn, F. S. (2020) · Minerva Access (University of Melbourne) · Cited by 6
Replication Redux: The Reproducibility Crisis and the Case of Deworming
Ozier, O. W. (2019) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-8835 · Cited by 4
“Putting a related idea into practice, coupling a measure of expert beliefs with a replication project is exactly what Camerer et al. (2016) did. Galiani, Gertler, and Romero’s main recommendation focuses on the way journals structure incentives. If journals require some combination of public data availability and data…”
InfluentialStrategic Intelligence: The Cognitive Capability to Anticipate Competitor BehaviorPreprint
Levine, S. S., & Nagel, R. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2258999 · Cited by 3
Forensic Science needs Registered ReportsPreprint
Chin, J. M., McFadden, R., & Edmond, G. (2019) · doi:10.31228/osf.io/yzt2q · Cited by 3
Screening $p$-Hackers: Dissemination Noise as BaitPreprint
Echenique, F., & He, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09164 · Cited by 3
“In a follow-up to the reproducibility crises in Psychology, Camerer et al. (2016) evaluate experiments in economics.”
BackgroundBertoldo, G., Callegher, C. Z., & Altoè, G. (2020) · doi:10.31234/osf.io/q9f86 · Cited by 3
“…published findings by promoting largescale replication efforts, where the protocol of a previous study is repeated as closely as possible with a new sample (Camerer et al., 2016; Camerer et al., 2018; Ebersole et al., 2016; Klein et al., 2014; Klein et al., 2018; Open Science Collaboration, 2015).”
BackgroundSignificance Testing in Natural and Biological Systems: A ReviewPreprint
Kak, S. C. (2021) · doi:10.36227/techrxiv.14599836.v1 · Cited by 3
“Efforts to replicate research in other fields such as behavioral economics [8], medicine [9], genetics [10] and neuroscience [11] likewise produce poor results.”
BackgroundWhat should comparative cognition expect from direct replication studies?Preprint
Farrar, B. G., & Clayton, N. S. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4urm8 · Cited by 3
Factor Investing in Sovereign Bond Markets: Deep Sample EvidencePreprint
Baltussen, G., Martens, M. P. E., & Penninga, O. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3873863 · Cited by 3
Interpreting the Will of the People: A Positive Analysis of Ordinal Preference Aggregation
Ambuehl, S., & Bernheim, B. D. (2021) · National Bureau of Economic Research · doi:10.3386/w29389 · Cited by 9
Participants’ Characteristics at Iser-Lab in 2020Preprint
Hanaki, N., Inukai, K., Masuda, T., & Shimodaira, Y. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3920755 · Cited by 6
The globalizability of temporal discountingPreprint
Ruggeri, K., Panin, A., Vdović, M., Većkalov, B., Abdul-Salaam, N., Achterberg, J., Akil, C., Amatya, J., Amatya, K., Andersen, T. L., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-1048207/v1 · Cited by 5
How failure to falsify in high-volume science contributes to the replication crisisPreprint
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2021) · doi:10.31234/osf.io/5fb7h · Cited by 3
Probabilistic forecasting of replication studiesPreprint
Pawel, S., & Held, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/fhwb7 · Cited by 4
New statistical metrics for multisite replication projectsPreprint
Mathur, M. B., & VanderWeele, T. J. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/w89s5 · Cited by 4
Simchi‐Levi, D. (2019) · Management Science · doi:10.1287/mnsc.2019.3553 · Cited by 3
Laypeople Can Predict Which Social Science Studies ReplicatePreprint
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/egw9d · Cited by 2
A Practical Guide to Registered Reports for EconomistsPreprint
Arpinon, T., & Espinosa, R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4110803 · Cited by 3
(FULL GUIDE) A Guide for Social Science Journal Editors on Easing into Open SciencePreprint
Silverstein, P., Elman, C., Montoya, A. K., McGillivray, B., Pennington, C. R., Harrison, C. H., Steltenpohl, C. N., Röer, J. P., Corker, K. S., Charron, L., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/hstcx · Cited by 4
Evaluating meta-analysis as a replication success measurePreprint
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., et al. (2023) · doi:10.31222/osf.io/ax825 · Cited by 3
Investigating Issues with Reproducibility in Journal Articles Published in Psychological SciencePreprint
Homewood, C. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6d5nq · Cited by 3
Conventional Wisdom, Meta-Analysis, and Research Revision in EconomicsPreprint
Gechert, S., Mey, B., Opatrný, M., et al. (2024) · doi:10.31222/osf.io/njd64 · Cited by 3
Questionable Research Practices and Open Science in Quantitative CriminologyPreprint
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2024) · CrimRxiv · doi:10.21428/cb6ab371.0d3ec44f · Cited by 3
AITurk: Using ChatGPT for Social Science ResearchPreprint
Qin, X., Huang, M., & Ding, J. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xkd23 · Cited by 4
Measuring Economic Preferences with Surveys and Behavioral ExperimentsPreprint
Kosfeld, M., Sharafi, Z., González, M. S., & Zou, N. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5114896 · Cited by 3
A Protocol for Structured Robustness Reproductions and Replicability AssessmentsPreprint
Peters, J., Brodeur, A., Dreber, A., Johannesson, M., Neubauer, F., & Rose, J. A. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5135862 · Cited by 4
“Following several large-scale replicability and computational reproducibility projects conducted over the past decade (Camerer et al. 2016, 2018; Chang & Li 2015; Open Science Collaboration 2015) , the boundary has recently been pushed to meta-robustness reproductions (Brodeur, Mikola, et al. 2024;…”
InfluentialSelf-citationThe Power of the Common Task FrameworkPreprint
Hellum, O., Jensen, T. I., Kelly, B. T., & Pedersen, L. H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5242901 · Cited by 4
Measuring Self-Reported Well-Being
Heffetz, O., & Caspi, Y. (2025) · National Bureau of Economic Research · doi:10.3386/w33847 · Cited by 3
Assessing the replicability of RCTs in RWE emulations
Köppe, J., Micheloud, C., Erdmann, S., Heyard, R., & Held, L. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02589-z · Cited by 4
Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects
Heyard, R., & Held, L. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327799 · Cited by 4
“To illustrate the applicability of the proposed methodology, we reanalyse data from the replication projects in Psychology [RPP, 1] and in Experimental Economics [RPEE, 2] provided by Altmejd et al. (2019) [31].”
Open science practices among early-career human-computer interaction researchers in the US
Chakravorti, T., Gautam, S., & Rajtmajer, S. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0334692 · Cited by 8
Revisiting big data optimism: risks of data-driven black box algorithms for society
Mahajan, S., & Helbing, D. (2026) · Ethics and Information Technology · doi:10.1007/s10676-026-09888-z · Cited by 9
The role of experiments for policy design
Werner, P., & Riedl, A. M. (2019) · Edward Elgar Publishing eBooks · doi:10.4337/9781788110563.00025 · Cited by 8
Transparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 3
Hall, L. M., & Hendricks, A. E. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03817-7 · Cited by 3
Esteemed Colleagues: A Model of the Effect of Open Data on Selective Reporting of Scientific Results
Spiegelman, E. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.761168 · Cited by 2
“The Open Science Collaboration (2015) successfully replicated 36 out of 100 experiments published in high-ranking psychology journals; Camerer et al. (2016, 2018) find reproducibility rates of around 61% in economics experiments.”
On the Accuracy of Replication Failure Rates
Schauer, J. M. (2022) · Multivariate Behavioral Research · doi:10.1080/00273171.2022.2066500 · Cited by 3
“…in social and behavioral psychology; the Experimental Economics Replication Project (EERP) ran replications of 18 different economics experiments (Camerer et al., 2016; Open Science Collaboration, 2015); and the Social Sciences Replication Project (SSRP) attempted to replicate 21 social science…”
BackgroundMethodSolidarity with Third Players in Exchange Networks: A Replication Study
Neuhofer, S., Schwaninger, M. C., Kittel, B., & Yang, G. (2022) · Jahrbuch für Handlungs- und Entscheidungstheorie · doi:10.1007/978-3-658-35878-5_5 · Cited by 2
Chan, E. W., Wong, J., Chan, C. S., & Cheung, F. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_18 · Cited by 4
Replication of "null results" - Absence of evidence or evidence of absence?
Pawel, S., Heyard, R., Micheloud, C., & Held, L. (2023) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.2305.04587 · Cited by 2
“…have been conducted in diverse fields, from the biomedical to the social sciences ( Prinz et al., 2011 ; Begley and Ellis, 2012 ; Klein et al., 2014 ; Open Science Collaboration, 2015 ; Camerer et al., 2016 , 2018 ; Klein et al., 2018 ; Cova et al., 2018 ; Errington et al., 2021 , among others).”
BackgroundA Prototype Hybrid Prediction Market for Estimating Replicability of Published Work
Chakravorti, T., Fraleigh, R. D., Fritton, T., McLaughlin, M., Singh, V. P., Griffin, C. H., Kwasnica, A. M., Pennock, D. M., Giles, C. L., & Rajtmajer, S. M. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230093 · Cited by 7
Li, T., Frank, K. A., & Chen, M. (2024) · Mathematics · doi:10.3390/math12030388 · Cited by 5
“The probabilities of reproducibility are driven by the reproducibility of a scholarly finding rather than its statistical significance, thus becoming tools for tackling the reproducibility crisis [60,61].”
BackgroundThe Alternative Factors Leading to Replication Crisis: Prediction and Evaluation
Chernov, G. (2024) · Evaluation Review · doi:10.1177/0193841x241229106 · Cited by 3
Reporting Standards, Certification/Accreditation, and Reproducibility
Simon, G., & Aliferis, C. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_17 · Cited by 2
The Limits of Empiricism: A Critique of Data-Driven Theory Development
Slyke, C. V., & Kamis, A. A. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3663682.3663689 · Cited by 2
“Even replications of psychology and economics experiments have been disappointing (Asendorpf et al., 2013; Camerer et al., 2016; Camerer et al., 2018; Klein et al., 2014).”
BackgroundScreening p -hackers: Dissemination noise as bait
Echenique, F., & He, K. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2400787121 · Cited by 5
“In a follow-up to the reproducibility crises in Psychology, Camerer et al. (2016) evaluate experiments in economics.”
BackgroundEconomics Peer-Review: Problems, Recent Developments, and Reform Proposals
Siemroth, C. (2023) · The American Economist · doi:10.1177/05694345241269484 · Cited by 7
“2 I will also not discuss the problems concerning p-hacking, publication bias, or non-reproducibility of empirical research (e.g., Brodeur et al., 2016; Brodeur et al., 2020; Camerer et al., 2016).”
The Language of (Non)Replicable Social Science
Herzenstein, M., Rosario, S., Oblander, S., & Netzer, O. (2024) · Psychological Science · doi:10.1177/09567976241254037 · Cited by 6
“…were replicated by Open Science Collaboration (2015; RPP), 49 by Many Labs (Ebersole et al., 2016; Klein et al., 2014, 2018, 2022; ML), 18 by Camerer et al. (2016; EE), 22 by Camerer et al. (2018; SSRP), 8 by Zwaan et al. (2018), and a range of individual replications that were…”
Toward Better Reporting in Oral Health Research
Jakubovics, N. S., & Schwendicke, F. (2024) · Journal of Dental Research · doi:10.1177/00220345241275459 · Cited by 4
“Large-scale replication studies in psychology and economics have found that replications often provide weaker evidence than the original studies or fail to reproduce the original findings and conclusions at all (Open Science Collaboration 2015; Camerer et al. 2016).”
Recovering the Anchoring of Economic Valuations
Guo, L. (2021) · American Economic Journal Microeconomics · doi:10.1257/mic.20200320 · Cited by 5
Preregistration and predictivism
Choi, H. H. (2024) · Synthese · doi:10.1007/s11229-024-04827-1 · Cited by 6
“Camerer et al. (2016) estimated 61% for economics but the replication effect sizes were on average only 66% that of the original (Camerer et al. 2016).”
On the Rival Nature of Data: Tech and Policy Implications
Gordon-Tapiero, A., Ligett, K., & Nissim, K. (2025) · doi:10.1145/3709025.3712211 · Cited by 2
Lindsay, R. M. (2025) · Advances in management accounting · doi:10.1108/s1474-787120250000037001 · Cited by 9
Replicability and Meta-Analysis
Schauer, J. M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_14 · Cited by 3
Counterfactuals with Experimental and Quasi-Experimental Variation
Battistin, E., & Bertoni, M. (2023) · Texts in quantitative political analysis · doi:10.1007/978-3-031-12982-7_3 · Cited by 3
Replicability of Experimental Data and Credibility of Economic Theory
Kandori, M. (2018) · Japanese Economic Review · doi:10.1111/jere.12175 · Cited by 2
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.