Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 601–700 of 1,372
Reis, D., & Friese, M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_5 · Cited by 14
Shaw, S. D., & Nave, G. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114129 · Cited by 8
Self-citationExternal Validity and Evidence Accumulation
Slough, T. L., & Tyson, S. A. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009375856 · Cited by 13
McCullough, B. D. (2018) · Economics · doi:10.5018/economics-ejournal.ja.2018-52 · Cited by 7
Cooperation among strangers with and without a monetary system
Bigoni, M., Camera, G., & Casari, M. (2020) · Edward Elgar Publishing eBooks · doi:10.4337/9781785363337.00016 · Cited by 7
“The overall consistency of results in experiments with money provides some comfort on the replicability of the empirical results (Camerer et al., 2016).”
Replication Redux: The Reproducibility Crisis and the Case of Deworming
Ozier, O. W. (2020) · The World Bank Research Observer · doi:10.1093/wbro/lkaa005 · Cited by 11
Effort Is Not a Monotonic Function of Skills: Results From a Global Mobile Experiment
Grabiszewski, K., & Horenstein, A. R. (2019) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2020.05.011 · Cited by 6
Larson, M. J. (2020) · Biofeedback · doi:10.5298/1081-5937-48.1.2 · Cited by 10
Microfinance RCTs in Development: Miracle or Mirage?
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0009 · Cited by 34
A framework for open policy analysis
de la Guardia, F. H., Grant, S., & Miguel, E. A. (2020) · Science and Public Policy · doi:10.1093/scipol/scaa067 · Cited by 19
“And third, the simple lack of reproducibility: the results of many studies simply cannot be replicated in comparable settings (Open Science Collaboration 2015; Camerer et al. 2016) , or even reproduced computationally using its original data and code (Chang and Li 2015; Gertler, Galiani, and Romero 2018).”
ResultMoffatt, P. G. (2021) · Foundations and Trends® in Econometrics · doi:10.1561/0800000035 · Cited by 9
Intrahousehold Consumption Allocation and Demand for Agency: A Triple Experimental Investigation
Afzal, U., D’Adda, G., Fafchamps, M., & Said, F. (2022) · American Economic Journal Applied Economics · doi:10.1257/app.20200672 · Cited by 10
Replication in experimental management accounting research - an editorial
Brüggen, A., Maas, V. S., & den Abbeele, A. G. H. L. V. (2021) · Journal of Management Control · doi:10.1007/s00187-021-00324-5 · Cited by 10
An advanced method to streamline p-hacking
Sarstedt, M., & Adler, S. J. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.113942 · Cited by 10
How redefining statistical significance can worsen the replication crisis
Williams, C. R. (2019) · Economics Letters · doi:10.1016/j.econlet.2019.05.007 · Cited by 9
Screen Recordings as a Tool to Document Computer Assisted Data Collection Procedures
Heycke, T., & Spitzer, L. (2019) · Psychologica Belgica · doi:10.5334/pb.490 · Cited by 10
Millroth, P., Nilsson, H., & Juslin, P. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500006161 · Cited by 13
What Crisis? Management Researchers’ Experiences with and Views of Scholarly Misconduct
Hopp, C., & Hoover, G. A. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-018-0079-4 · Cited by 14
Corgnet, B., DECK, C. A., DeSantis, M., Hampton, K. W., & Kimbrough, E. O. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3387603 · Cited by 8
Generalizability is not optional: Insights from a cross-cultural study of social discountingPreprint
Tiokhin, L., Hackman, J., Munira, S., Jesmin, K., & Hruschka, D. J. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/f5s84 · Cited by 10
Open Science is Liberating and Can Foster CreativityPreprint
Frankenhuis, W. E., & Nettle, D. (2018) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/edhym · Cited by 9
Transparency of CHI Research Artifacts: Results of a Self-Reported SurveyPreprint
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3bu6t · Cited by 11
Brodeur, A., Cook, N., & Heyes, A. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/a9vhr · Cited by 10
Reproducibility in Management SciencePreprint
Fišar, M., Greiner, B., Huber, C., Katok, E., Özkes, A., & Management Science Reproducibility Collaboration (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mydzv · Cited by 11
Kirchler, M., & Weitzel, U. (2023) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2023.106975 · Cited by 9
Self-citationPrice formation in field prediction markets: The wisdom in the crowd
Bossaerts, F., Yadav, N., Bossaerts, P. L., Nash, C., Todd, T., Rudolf, T., Hutchins, R., Ponsonby, A., & Mattingly, K. (2024) · Journal of Financial Markets · doi:10.1016/j.finmar.2023.100881 · Cited by 11
Registered report adoption in academic journals: assessing rates in different research domains
Lin, T., Cheng, H., Cheng, L.-F., & Hung, T. (2024) · Scientometrics · doi:10.1007/s11192-023-04896-y · Cited by 14
“Four large-scale replication projects have been published, focusing on psychology (Open Science Collaboration, 2015), social science (Camerer et al., 2018), medicine (Errington et al., 2021), and economics (Camerer et al., 2016), respectively.”
Mundinger, C., Schulz, N. K. E., Singh, P., Janz, S., Schurig, M., Seidemann, J., Kurtz, J., Müller, C., Schielzeth, H., von Kortzfleisch, V. T., et al. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003019 · Cited by 9
“Whether effect sizes or statistical significance are easier to reproduce appears to be field-specific (in psychology: 47% versus 36% [6]; in economics: 61% versus 67% [10]).”
Common method variance (CMV) bias: its implications, how to detect it, and how to handle it
Andersen, R., Thomakos, D. D., & Wood, G. (2025) · Human Resource Development International · doi:10.1080/13678868.2025.2537161 · Cited by 10
Replication of Audit and Financial Accounting Research: We Do More than We Think
Salterio, S. E., Luo, Y., & Adamson, C. (2022) · Accounting Horizons · doi:10.2308/horizons-2022-152 · Cited by 12
Hansson, S., Larsson, U., & Walve, J. (2019) · Estuarine Coastal and Shelf Science · doi:10.1016/j.ecss.2019.106261 · Cited by 8
Lanz, B., & Reins, E. (2020) · The Energy Journal · doi:10.5547/01956574.42.4.blan · Cited by 8
“Note also that the results from this experimental procedure were successfully replicated by Camerer et al. (2016).”
ResultMultidimensional social identity and redistributive preferences: an experimental study
Hong, F., Riyanto, Y. E., & Zhang, R. (2021) · Theory and Decision · doi:10.1007/s11238-021-09834-z · Cited by 12
Illusion of control in farmers’ investment and financing decisions
Labajova, K., Höhler, J., Lagerkvist, C. J., Müller, J., & Rommel, J. (2021) · Agricultural Finance Review · doi:10.1108/afr-09-2020-0140 · Cited by 15
Happiness in the Lab: What Can Be Learned About Subjective Well-Being from Experiments?
Ifcher, J., Zarghamee, H., & Goff, S. (2021) · Springer eBooks · doi:10.1007/978-3-319-57365-6_187-1 · Cited by 6
Long, J. A. (2021) · Annals of the International Communication Association · doi:10.1080/23808985.2021.1979421 · Cited by 11
Luo, T., Meng, R., Wang, X., & Liu, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.305 · Cited by 10
The p-Value and Statistical Significance Testing
Hirschauer, N., Grüner, S., & Mußhoff, O. (2022) · SpringerBriefs in applied statistics and econometrics · doi:10.1007/978-3-030-99091-6_6 · Cited by 11
The intrinsic value of decision rights: A replication and an extension to team decision making
Buffat, J., Praxmarer, M., & Sutter, M. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.03.019 · Cited by 6
A plea for preregistration in taxonomy
Conix, S., Cuypers, V., Zachos, F. E., Artois, T., & Monnens, M. (2023) · Megataxa · doi:10.11646/megataxa.10.1.1 · Cited by 7
Fecher, B., & Wagner, G. G. (2016) · Science · doi:10.1126/science.351.6280.1405-b · Cited by 7
Quantifying Research Integrity
Seadle, M. S. (2016) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00743ed1v01y201611icr053 · Cited by 23
Forensic science needs registered reports
Chin, J. M., McFadden, R., & Edmond, G. (2019) · Forensic Science International Synergy · doi:10.1016/j.fsisyn.2019.10.005 · Cited by 8
Stability of Experimental Results: Forecasts and Evidence
DellaVigna, S., & Pope, D. G. (2019) · American Economic Journal Microeconomics · doi:10.1257/mic.20200129 · Cited by 23
Predicting Bond Returns: 70 Years of International Evidence
Baltussen, G., Martens, M. P. E., & Penninga, O. (2021) · Financial Analysts Journal · doi:10.1080/0015198x.2021.1908775 · Cited by 8
Open Science, Replicability, and Transparency in Modelling
Prike, T. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7_10 · Cited by 12
(2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0441-1 · Cited by 7
Matheson, G. J., Plavén‐Sigray, P., Tuisku, J., Rinne, J. O., Matuskey, D., & Červenka, S. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04541-y · Cited by 8
The assessment of replication success based on relative effect sizePreprint
Held, L., Micheloud, C., & Pawel, S. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-196008 · Cited by 7
Normatively Framed Relative Performance Feedback Field Experiment and Replication
Brade, R., Himmler, O., & Jäckle, R. (2018) · AEA Randomized Controlled Trials · doi:10.1257/rct.3288 · Cited by 10
Increasing the reproducibility of research will reduce the problem of apophenia (and more)
Jones, P. M., & Martin, J. E. (2021) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-021-02006-1 · Cited by 8
Forking Paths in Empirical StudiesPreprint
Coqueret, G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.3999379 · Cited by 12
Improving transparency in observational social science research: A pre-analysis plan approachPreprint
Burlig, F. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/qemkz · Cited by 9
Reproduce to validate: A comprehensive study on the reproducibility of economics research
Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2024) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/caje.12728 · Cited by 42
Fink, L., & Marcus, J. (2024) · Economic Inquiry · doi:10.1111/ecin.13267 · Cited by 16
Vigne, N. G. L. (2024) · Journal of Experimental Criminology · doi:10.1007/s11292-024-09650-9 · Cited by 6
“…that rigorous evaluations, which she defines solely as RCTs, frequently yield null results and that interventions previously found to be effective often fail to replicate, a prevalent problem in other social science disciplines as well (Camerer et al., 2016; Open Science Collaboration, 2015).”
Generalizability of choice architecture interventions
Szászi, B., Goldstein, D. G., Soman, D., & Michie, S. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00471-9 · Cited by 21
Langley, P. C. (2017) · INNOVATIONS in pharmacy · doi:10.24926/21550417.1368 · Cited by 6
Statistical power and productivity effects of transport investments: A critical review
Tveter, E., & Holmgren, J. (2024) · Research in Transportation Economics · doi:10.1016/j.retrec.2024.101430 · Cited by 7
Let’s talk about language—and its role for replicability
Schmalz, X., Breuer, J., Haim, M., Hildebrandt, A., Knöpfle, P., Leung, A. Y., & Roettger, T. B. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04381-2 · Cited by 9
“…replicability, in its broad sense, has recently been identi fi ed for large sets of studies in psychology (Open Science Collaboration, 2015; Klein et al. 2022), medicine (Ioannidis, 2005), economics (Camerer et al. 2016), and the behavioural and social sciences more generally (Camerer et al. 2018).”
Poor hypotheses and research waste in biology: learning from a theory crisis in psychology
Nakagawa, S., Armitage, D. W., Froese, T., Yang, Y., & Lagisz, M. (2025) · BMC Biology · doi:10.1186/s12915-025-02134-w · Cited by 15
A non-parametric significance test to compare corpora
Koplenig, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0222703 · Cited by 10
“It is part of an ongoing lively debate among statisticians and quantitative researchers [3–10] to find ways to ‘fix statistics’ [11] (partly) in response to the current reproducibility crisis [12–16] in psychology [17,18], experimental economics [19] and social science [20].”
BackgroundCriticism Of Behavioural Economics: Attacks Towards Ideology, Evidence And Practical Application
Pūce, L. (2019) · Journal of WEI Business and Economics · doi:10.36739/jweibe.2019.v8.i1.3 · Cited by 9
Sample Size Calculation in Economic Experiments
Grüner, S. (2020) · Jahrbücher für Nationalökonomie und Statistik · doi:10.1515/jbnst-2019-0020 · Cited by 11
Reliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
How often are basic details of the research process mentioned in social science research papers?
Coates, A. (2020) · Learned Publishing · doi:10.1002/leap.1330 · Cited by 8
“For example, the ‘replication crisis’ found that the results of many studies in social science fail to replicate (Camerer et al., 2016; Open Science Collaboration, 2015; Yom, 2018).”
BackgroundTrials and Tribulations: The Rise and Fall of the RCT in the WASH Sector
Spears, D., Ban, R., & Cumming, O. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0008 · Cited by 26
Barreiro‐Hurlé, J. (2021) · Spanish Journal of Agricultural Research · doi:10.5424/sjar/2021194-18956 · Cited by 7
“io/initiatives/prereg Following the experience of other fields of science economics has been subject to a heated debate on the issue of replicability, selective reporting and publication bias (Camerer et al., 2016; Nosek et al., 2018; Kvarven et al., 2020). One of the recommendations put forward to improve research ef…”
BackgroundInfluentialCamerer, C. F. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2216020119 · Cited by 9
Self-citationDistrust: Big Data, Data-Torturing, and the Assault on Science
Smith, G. (2023) · doi:10.1093/oso/9780192868459.001.0001 · Cited by 12
Cognitive reflection and 2D:4D: Evidence from a large population sample
Neyse, L., Fossen, F. M., Johannesson, M., & Dreber, A. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.03.020 · Cited by 6
Self-citationSegerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., Tomiyama, A. J., Bacon, S., Bennett, G. G., Brondolo, E. N., Czajkowski, S. M., Davidson, K. W., et al. (2023) · Annals of Behavioral Medicine · doi:10.1093/abm/kaac044 · Cited by 14
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Health Psychology · doi:10.1037/hea0001236 · Cited by 7
The Replication of Equivalence Studies
Micheloud, C., & Held, L. (2024) · Biometrical Journal · doi:10.1002/bimj.202300232 · Cited by 7
“Various empirical domains of science are involved in these replication efforts: psychology (Open Science Collaboration 2015), social sciences (Camerer et al. 2016), economics (Camerer et al. 2018), and more recently cancer biology (Errington et al. 2021), among others.”
Sign-Congruence, External Validity, and Replication
Slough, T. L., & Tyson, S. A. (2025) · Political Analysis · doi:10.1017/pan.2024.26 · Cited by 11
“…and is well-suited to questions about publication bias or researcher integrity.14 It is important to consider the relationship between Proposition 1 and replication designs that leverage replications of multiple distinct studies (for examples in economics, see Camerer et al., 2016, 2018).”
One-shot reciprocity under error management is unbiased and fragile
Zimmermann, J., & Efferson, C. M. (2016) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2016.06.005 · Cited by 4
“Given recent studies showing that experimental results on social behavior do not replicate as often as we might like (Camerer et al., 2016; Open-Science-Collaboration, 2015; Shanks et al., 2013), we should approach mixed empirical results with some skepticism, and future experimental research on…”
Human Resource Management and Nudging: An Experimental Analysis on Goal Settings in German Companies
Grunewald, M., Hammermann, A., & Placke, B. (2017) · International Journal of Economics and Finance · doi:10.5539/ijef.v9n9p147 · Cited by 6
“Camerer et al. (2016) conducted worldwide replications from particularly well-known behavioural economics studies, with the result that 78 percent of the research findings could be replicated and the rest of the findings just shortly missed the level of significance.”
BackgroundImproving Reproducible Deep Learning Workflows with DeepDIVA
Alberti, M., Pondenkandath, V., Vögtlin, L., Würsch, M., Ingold, R., & Liwicki, M. (2019) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds.2019.00-14 · Cited by 6
“This problem – also known as the reproducibility crisis or the replication crisis – has been extensively studied in the fields of psychology [1], economics [2] and medicine [3].”
BackgroundDoes the privacy paradox exist? Comment on Yu et al.’s (2020) meta-analysis
Dienlin, T., & Sun, Y. (2021) · Meta-Psychology · doi:10.15626/mp.2020.2711 · Cited by 9
The Limitations of Social Science as the Arbiter of Blame: An Argument for Abandoning Retribution
Tullett, A. M. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211033284 · Cited by 8
“For example, studies that were not preregistered, have small samples, and report p values close to the .05 cutoff for statistical significance are more likely to be false positives (Camerer et al., 2016; Forsell et al., 2019; Simmons et al., 2020).”
BackgroundFanelli, D. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0004 · Cited by 7
Examining the Reproducibility of 6 Published Studies in Public Health Services and Systems Research
Harris, J. K., Wondmeneh, S. B., Zhao, Y., & Leider, J. P. (2018) · Journal of Public Health Management and Practice · doi:10.1097/phh.0000000000000694 · Cited by 8
“While most of the work on reproducibility in science has been in the biomedical or psychology fields, 1 recent paper in the journal Science examined economics papers and found that 61% (11 of 18) papers were reproducible.(13) Less has been done to examine the reproducibility of public health research.”
BackgroundA note on replication analysis
Hoeppner, S. (2019) · International Review of Law and Economics · doi:10.1016/j.irle.2019.05.004 · Cited by 5
Replications in economic psychology and behavioral economics
Schultze, T., Huber, J., Kirchler, M., & Mojzisch, A. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2019.102199 · Cited by 6
Self-citationAnvari, F., Olsen, J., Yiu, H. W., & Feldman, G. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104052 · Cited by 9
Half a century of Lighting Research & Technology: A bibliometric review
Chen, Q., Ru, T., Zhai, D., Huang, X., Li, Y., Qian, L., Wang, Y., & Zhou, G. (2019) · Lighting Research & Technology · doi:10.1177/1477153519857788 · Cited by 6
Editorial: Replicability in Cognitive Science
Strickland, B., & Cruz, H. D. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-021-00531-y · Cited by 7
Open Science and Educational Technology Research
Lodge, J. M., Corrin, L., Hwang, G., & Thompson, K. (2021) · Australasian Journal of Educational Technology · doi:10.14742/ajet.7565 · Cited by 6
“While the initial focus was on the replication of findings in social and experimental psychology, other systemic issues emerged and the problem began to be discussed in other disciplines (for example, in Medicine; Mobley et al., 2013, in Economics; Camerer et al., 2016).”
BackgroundThe globalizability of temporal discountingPreprint
Ruggeri, K., Panin, A., Vdović, M., Većkalov, B., Abdul-Salaam, N., Achterberg, J., Amatya, J., Amatya, K., Arunasalam, A., Ashcroft-Jones, S., et al. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2enfz · Cited by 8
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2022) · doi:10.31234/osf.io/dj3xe · Cited by 6
Estimating the extent of inflated significance in economicsPreprint
Bruns, S. B., Deressa, T. K., Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2022) · doi:10.31222/osf.io/h29xn · Cited by 4
Comparing Meta-Analyses and Pre-Registered Multiple Labs Replication ProjectsPreprint
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/brzwt · Cited by 6
Self-citationYang, Y., Hillebrand, H., Lagisz, M., et al. (2021) · doi:10.32942/osf.io/kdq45 · Cited by 6
Heterogeneity of Beliefs and Trading Behavior — A ReexaminationPreprint
Füllbrunn, S., Huber, C., Eckel, C. C., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4141893 · Cited by 6
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., et al. (2023) · doi:10.31222/osf.io/fhdbs · Cited by 6
The Replication Database: Documenting the Replicability of Psychological SciencePreprint
Röseler, L., Kaiser, L., Doetsch, C. A., Klett, N., Seida, C., Schütz, A., Aczél, B., Adelina, N., Agostini, V., Alarie, S., et al. (2024) · doi:10.31222/osf.io/me2ub · Cited by 6
Individual Preferences for Truth-Telling
Schudy, S., Grundmann, S., & Spantig, L. (2026) · CESifo · doi:10.65864/zgqjzplwf5 · Cited by 6
Characterisation and calibration of multiversal methods
Cantone, G. G., & Tomaselli, V. (2024) · Advances in Data Analysis and Classification · doi:10.1007/s11634-024-00610-9 · Cited by 8
“For years, academic journals consistently published results that were unreliable at best, and never replicable at worst (OPEN Science COLLABORATION 2015; Baker 2016; Camerer et al. 2016; Carlisle 2017; Pridemore et al. 2018; Stroebe 2019; Gordon et al. 2020; van Zwet and Cator 2021).”
Chakravorti, T., Koneru, S., & Rajtmajer, S. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319334 · Cited by 16
“Large-scale replication projects in psychology [23], economics [24], sociology [25], biology [26] and beyond have turned up disappointing results.”
Wang, D., & Liu, L. (2020) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3383583.3398500 · Cited by 9
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.