Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 301–400 of 1,372
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · National Bureau of Economic Research · doi:10.3386/w27554 · Cited by 53
“61 Camerer et al. (2016) replicated 18 laboratory experiments published in the American Economic Review (AER) and”
The illusion of science in comparative cognitionPreprint
Farrar, B. G., & Ostojić, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hduyx · Cited by 31
Vilhuber, L. (2019) · AEA Papers and Proceedings · doi:10.1257/pandp.109.718 · Cited by 33
Ethics of RCTs: Should Economists Care about Equipoise?
Abramowicz, M., & Szafarz, A. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0012 · Cited by 56
Fitrianie, S., Bruijnes, M., Abdulrahman, A., & Brinkman, W. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103482 · Cited by 38
The Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
“On the other hand, the replication initiative by Camerer et al. (2016) provides evidence that experimental economics fares relatively well regarding replicability.”
Rieger, M. O., Wang, M., & Hens, T. (2020) · PLoS ONE · doi:10.1371/journal.pone.0245692 · Cited by 28
“A priori, it is not clear whether these methods measure the same concept–an important and unfortunately often overlooked issue [10]–although occasionally, relations between different methodologies have been found [6].”
BackgroundKasy, M. (2021) · The Journal of Economic Perspectives · doi:10.1257/jep.35.3.175 · Cited by 42
Witt, J. K. (2019) · Meta-Psychology · doi:10.15626/mp.2018.895 · Cited by 29
“Such a shift could help guard against practices that have contributed to recent failures to replicate in various scientific fields (Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundWhich Findings Should Be Published?
Frankel, A. P., & Kasy, M. (2022) · American Economic Journal Microeconomics · doi:10.1257/mic.20190133 · Cited by 55
Experimental and behavioral economics to inform agri-environmental programs and policies
Palm‐Forster, L. H., & Messer, K. D. (2021) · Handbook of agricultural economics · doi:10.1016/bs.hesagr.2021.10.006 · Cited by 52
Fraser, H. S., Bush, M., Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Gould, E., Hemming, V., Hamilton, D. G., Rumpff, L., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0274429 · Cited by 43
RCTs in Development Economics, Their Critics and Their Evolution
Ogden, T. N. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0006 · Cited by 34
Forecasting the Results of Experiments: Piloting an Elicitation Strategy
DellaVigna, S., Otis, N. G., & Vivalt, E. (2020) · AEA Papers and Proceedings · doi:10.1257/pandp.20201080 · Cited by 25
Krefeld-Schwalb, A., & Scheibehenne, B. (2022) · Marketing Letters · doi:10.1007/s11002-022-09662-3 · Cited by 31
Thắng, P. C., & Nguyet, T. T. T. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e12747 · Cited by 37
“Camerer et al. [41,42] argue for the need to test and retest the UTAUT model believe that it should be tested in different fields and contexts to modify this model.”
BackgroundPolitical ideology and trust in scientists in the USA
Gligorić, V., Kleef, G. A. V., & Rutjens, B. T. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02147-z · Cited by 38
“Although the replication crisis in both natural and social sciences 63–65 is fairly recent (compared with, for example, when the lower trust among conservatives was first observed 2 ), it can indeed lead to public’s decreased trust in science 66–68 .”
Anderson, M. H., & Lemken, R. K. (2019) · Journal of Management Studies · doi:10.1111/joms.12527 · Cited by 25
New Evidence on Information Disclosure through Restaurant Hygiene Grading
Ho, D. E., Ashwood, Z. C., & Handan‐Nader, C. (2019) · American Economic Journal Economic Policy · doi:10.1257/pol.20180230 · Cited by 31
Stock-specific sentiment and return predictability
Coqueret, G. (2020) · Quantitative Finance · doi:10.1080/14697688.2020.1736314 · Cited by 26
Do Monetary Incentives Undermine Performance on Intrinsically Enjoyable Tasks? A Field Test
Esteves‐Sorenson, C., & Broce, R. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_00947 · Cited by 38
Corgnet, B., DECK, C. A., DeSantis, M., Hampton, K., & Kimbrough, E. O. (2019) · Management Science · doi:10.1287/mnsc.2022.4463 · Cited by 33
Ellis, R. J. (2022) · eNeuro · doi:10.1523/eneuro.0017-22.2022 · Cited by 30
“…2019; Open Science Collaboration, 2015; Ebersole et al., 2016), cancer biology (Prinz et al., 2011; Begley and Ellis, 2012; Errington et al., 2021), social sciences (Camerer et al., 2018), economics (Camerer et al., 2016), and machine learning (Beam et al., 2020; Pineau et al., 2020; Table 1).”
BackgroundPromoting Reproducibility and Replicability in Political Science
Brodeur, A., Esterling, K., Peters, J., Bueno, N., DESPOSATO, S. W., Dreber, A., Genovese, F., GREEN, D. P., Hepplewhite, M., de la Guardia, F. H., et al. (2024) · Research & Politics · doi:10.1177/20531680241233439 · Cited by 42
“A few large-scale replication projects have taken place recently, including one in psychology (Open Science Collaboration, 2015), one in experimental economics (Camerer, 2016) and a social science replication project (Camerer, 2018).”
BackgroundSelf-citationGlöckner, A., Fiedler, S., & Renkewitz, F. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000384 · Cited by 28
How Credible Is Trade Union Research? Forty Years of Evidence on the Monopoly–Voice Trade-Off
Doucouliagos, H., Freeman, R. B., Laroche, P., & Stanley, T. D. (2017) · Industrial and Labor Relations Review · doi:10.1177/0019793917751144 · Cited by 36
LaPlante, D. A. (2019) · International Gambling Studies · doi:10.1080/14459795.2019.1672768 · Cited by 22
“marketing) also have identified similarly troubling replication rates or concerns (Berman, Pekelis, Aisling, & Van den Bulte, 2018; Camerer et al., 2016; Hutson, 2018; Kaiser, 2017).”
BackgroundGlobal Factor PremiumsPreprint
Baltussen, G., Swinkels, L. A. P., & Vliet, P. V. A. D. T. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3325720 · Cited by 24
Meta-Analysis of Inequality Aversion EstimatesPreprint
Nunnari, S., & Pozzi, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4169385 · Cited by 31
Mass Reproducibility and Replicability: A New HopePreprint
Brodeur, A., Mikola, D., & Cook, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790780 · Cited by 72
Flynn, K. C., Frazier, A. E., & Admas, S. (2020) · Remote Sensing · doi:10.3390/rs12182867 · Cited by 26
Page, L., NOUSSAIR, C. N., & Slonim, R. L. (2021) · Journal of the Economic Science Association · doi:10.1007/s40881-021-00107-7 · Cited by 32
Skating on Thin Evidence: Implications for Public Policy
Doucouliagos, H., Paldam, M., & Stanley, T. D. (2018) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2018.03.004 · Cited by 24
The need for open science practices and well-conducted replications in the field of gambling studies
Wohl, M. J. A., Tabri, N., & Zelenski, J. M. (2019) · International Gambling Studies · doi:10.1080/14459795.2019.1672769 · Cited by 23
“Unfortunately, there is reason to believe that many fields in the social sciences have a weak rate of replicability (Camerer et al., 2016, 2018; Open Science Collaboration, 2015), a revelation that has led to the so-called ‘replication crisis’ or ‘credibility revolution’.”
BackgroundThe Pied Piper: Prizes, Incentives, and Motivation Crowding-in
Bruni, L., Pelligra, V., Reggiani, T., & Rizzolli, M. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04154-3 · Cited by 33
Justify Your Alpha: A Primer on Two Practical ApproachesPreprint
Maier, M., & Lakens, D. (2021) · doi:10.31234/osf.io/ts4r6 · Cited by 23
A Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 34
“These indicators have been used in systematic replication studies (Open Science Collaboration, 2015; Camerer et al. 2016, 2018).”
Self-citationThe power of prediction markets
Mann, A. (2016) · Nature · doi:10.1038/538308a · Cited by 22
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 23
“…all estimates may not generalize, publication bias may still be of similar magnitude in other fields; but specific fields with a higher proportion of published negative evidence and/or a higher demonstrated reproducibility (Camerer et al., 2016) are likely to be less affected by publication bias.”
BackgroundCash versus extra-credit incentives in experimental asset markets
Ding, S., Volodymyr Lugovskyy, A., Daniela Puzzello, B., Tucker, S. J., & Williams, A. W. (2018) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2018.03.014 · Cited by 21
Tahamont, S., Jelveh, Z., Chalfin, A. J., Yan, S., & Hansen, B. (2020) · Journal of Quantitative Criminology · doi:10.1007/s10940-020-09461-x · Cited by 25
“Indeed this form of publication bias (filing papers in the desk drawer) is thought to have contributed to the “replication crisis” in empirical social science (Camerer et al. 2016; Gilbert et al. 2016; Pridemore et al. 2018; Vivalt 2017).”
BackgroundMalmendier, U. M. (2018) · Handbook of behavioral economics · doi:10.1016/bs.hesbe.2018.08.001 · Cited by 60
Reimagining peer review as an expert elicitation process
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A. M., Hemming, V., Wintle, B. C., Burgman, M. A., & Fidler, F. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06016-0 · Cited by 26
A Practical Guide to Registered Reports for Economists
Arpinon, T., & Espinosa, R. (2022) · Journal of the Economic Science Association · doi:10.1007/s40881-022-00123-1 · Cited by 39
Promoting pro-environmental choices while addressing energy poverty
Valle, N. D., D’Arcangelo, C., & Faillo, M. (2024) · Energy Policy · doi:10.1016/j.enpol.2023.113967 · Cited by 23
Preregistration and reproducibility
Strømland, E. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2019.01.006 · Cited by 27
Situational experience around the world: A replication and extension in 62 countries
Lee, D. I., Gardiner, G., Baranski, E. N., Members of The International Situations Project, & Funder, D. C. (2020) · Journal of Personality · doi:10.1111/jopy.12558 · Cited by 23
“Extending to fields outside of psychology, (Button et al., 2013; Camerer et al., 2016; Cova et al., in press) replication projects have provided insights into what we know, or what we thought we knew, about human behavior (Simmons, Nelson, & Simonsohn, 2011; Open Science Collaboration, 2015).”
BackgroundIntroduction: Randomization in the Tropics Revisited, a Theme and Eleven Variations
Deaton, A. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0002 · Cited by 29
Advancing the Cognitive Science of Religion through Replication and Open Science
Hoogeveen, S., & van Elk, M. (2021) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.39039 · Cited by 23
Ben-David, D., & Sade, O. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3361710 · Cited by 32
God Insures those Who Pay? Formal Insurance and Religious Offerings in Ghana*
Auriol, E., Lassébie, J., Panin, A., Raiber, E., & Seabright, P. (2020) · The Quarterly Journal of Economics · doi:10.1093/qje/qjaa015 · Cited by 92
“We can therefore examine separately the results for wave 1 and wave 2, and see to what extent they fulfil one of the four criteria set out by (Camerer et al., 2016) for successful replication.”
ResultPsychology’s reproducibility problem is exaggerated – say psychologists
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.19498 · Cited by 21
Supporting Formulary Decisions: The Discovery of New Facts or Constructed Evidence?
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.438 · Cited by 16
Power(ful) guidelines for experimental economists
Vasilaky, K., & Brock, J. (2020) · Journal of the Economic Science Association · doi:10.1007/s40881-020-00090-5 · Cited by 22
Friendship and partner choice in rural ColombiaPreprint
Redhead, D., Ragione, A. D., & Ross, C. T. (2022) · PsyArXiv (OSF Preprints) · doi:10.1016/j.evolhumbehav.2022.08.004 · Cited by 34
Field testing the transferability of behavioural science knowledge on promoting vaccinations
Saccardo, S., Dai, H., Han, M. A., Vangala, S., Hoo, J., & Fujimoto, J. T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01813-4 · Cited by 29
Conservative Apostles of Objectivity and the Myth of a “Liberal Bias” in Science
Larrègue, J. (2017) · The American Sociologist · doi:10.1007/s12108-017-9366-9 · Cited by 21
Randomization Inference for Accounting Researchers
White, R. M., & Webb, M. D. (2021) · Journal of Financial Reporting · doi:10.2308/jfr-2021-006 · Cited by 22
Funding of Clinical Trials and Reported Drug Efficacy
Oostrom, T. (2024) · Journal of Political Economy · doi:10.1086/730383 · Cited by 30
Variability in the analysis of a single neuroimaging dataset by many teamsPreprint
Botvinik‐Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/843193 · Cited by 21
Self-citationPredicting the Replicability of Social Science Lab ExperimentsPreprint
Altmejd, A., Dreber, A., Forsell, E., Ho, T., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/zamry · Cited by 23
Self-citationThe Rhetorical Superiority of Poor Economics
Labrousse, A. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0010 · Cited by 21
A FAIR Case for a Live Computational Commons
Omar, C., Coblenz, M. J., & Madhavapeddy, A. (2025) · doi:10.1145/3759536.3763802 · Cited by 20
Trails of Data: Three Cases for Collecting Web Information for Social Science Research
Li, F., Zhou, Y., & Cai, T. (2019) · Social Science Computer Review · doi:10.1177/0894439319886019 · Cited by 26
“3 In recent years, social scientists from multiple disciplines have launched a debate about reproducibility (Anderson et al., 2016; Camerer et al., 2016; Christensen & Miguel, 2016; Gilbert, King, Pettigrew, & Wilson, 2016).”
BackgroundBrade, R., Himmler, O., & Jäckle, R. (2022) · Economics of Education Review · doi:10.1016/j.econedurev.2022.102268 · Cited by 27
I enjoy hurting my classmates: On the relation of boredom and sadism in schools
Pfattheicher, S., Lazarević, L. B., Nielsen, Y. A., Westgate, E. C., Krstić, K., & Schindler, S. (2022) · Journal of School Psychology · doi:10.1016/j.jsp.2022.10.008 · Cited by 29
Naaman, K., Grant, S., Kianersi, S., Supplee, L. H., Henschel, B., & Mayo-Wilson, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.221093 · Cited by 30
The wisdom of many in few: Finding individuals who are as wise as the crowd.
Himmelstein, M., Budescu, D. V., & Ho, E. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001340 · Cited by 25
Reconsidering, refashioning, and reconceptualizing research methodology in international business
Delios, A., Welch, C., Nielsen, B. B., Aguinis, H., & Brewster, C. J. (2023) · Journal of World Business · doi:10.1016/j.jwb.2023.101488 · Cited by 28
Replication games: how to make reproducibility research more systematic
Brodeur, A., Dreber, A., de la Guardia, F. H., & Miguel, E. A. (2023) · Nature · doi:10.1038/d41586-023-02997-5 · Cited by 18
Self-citationEthics and international business research: Considerations and best practices
Miller, S. R., Moore, F., & Eden, L. (2023) · International Business Review · doi:10.1016/j.ibusrev.2023.102207 · Cited by 35
A global test of MPA spillover benefits to recreational fisheries
Franceschini, S., Lynham, J. M., & Madin, E. M. P. (2024) · Science Advances · doi:10.1126/sciadv.ado9783 · Cited by 21
Innovation Policy in a Networked World
Sorenson, O. (2017) · Innovation Policy and the Economy · doi:10.1086/694407 · Cited by 21
Distance from a distance: the robustness of psychological distance effects
Trautmann, S. T. (2019) · Theory and Decision · doi:10.1007/s11238-019-09696-6 · Cited by 26
Hedges, L. V. (2019) · Methodology · doi:10.1027/1614-2241/a000173 · Cited by 24
Open Science Practices in Gambling Research Publications (2016–2019): A Scoping Review
Louderback, E. R., Gainsbury, S., Heirene, R. M., Amichia, K., Grossman, A., Bernhard, B. J., & LaPlante, D. A. (2022) · Journal of Gambling Studies · doi:10.1007/s10899-022-10120-y · Cited by 26
“Many behavioral science research fields have surprisingly low likelihoods of replicating key findings (Camerer et al., 2016, 2018; Nosek & Errington, 2020; Nosek & Lakens, 2014; Open Science Collaboration, 2012, 2015) and struggle with publication bias (i.e., a preference for publishing…”
Ultra-Chaos: An Insurmountable Objective Obstacle of Reproducibility and Replication
Liao, S., & Qin, S. (2022) · Advances in Applied Mathematics and Mechanics · doi:10.4208/aamm.oa-2021-0364 · Cited by 22
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22193 · Cited by 31
Langley, P. C., & Rhee, T. G. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.439 · Cited by 17
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.436 · Cited by 15
Radžvilas, M., Pretis, F. D., Peden, W., Tortoli, D., & Osimani, B. (2022) · Computational Economics · doi:10.1007/s10614-022-10250-w · Cited by 21
Can large language models help predict results from a complex behavioural science study?
Lippert, S., Dreber, A., Johannesson, M., Tierney, W., Cyrus-Lai, W., Uhlmann, E. L., Emotion Expression Collaboration, & Pfeiffer, T. (2024) · Royal Society Open Science · doi:10.1098/rsos.240682 · Cited by 45
“…sizes and their statistical significance have been elicited from experts in several studies, most notably for large-scale replication projects [1,6–8], but also for studies investigating the generalizability of research claims and the applicability of competing theories to novel empirical…”
BackgroundSelf-citationShow Us Your Data: Connect the Dots, Improve Science
Levine, S. S. (2018) · Management and Organization Review · doi:10.1017/mor.2018.19 · Cited by 19
Bull, C., Courty, P., Doyon, M., & Rondeau, D. (2019) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2019.01.003 · Cited by 18
The Dynamics of Retraction in Epistemic Networks
LaCroix, T., Geil, A., & O’Connor, C. (2020) · Philosophy of Science · doi:10.1086/712817 · Cited by 25
Mayo‐Wilson, E., Grant, S., & Supplee, L. H. (2021) · Prevention Science · doi:10.1007/s11121-021-01284-x · Cited by 27
“However, studies using these methods often cannot be reproduced, which may be attributable to other limitations related to transparency and openness (Camerer et al., 2016, 2018; Open Science Collaboration, 2015).”
BackgroundSterck, O. C. (2019) · Journal of Economic Surveys · doi:10.1111/joes.12330 · Cited by 17
“…in applied economics (see e.g., Brodeur et al. (2016); Franco et al. (2014) on p-value hacking and publication bias; Maniadis et al. (2017) or Camerer et al. (2016) on replicability; Pritchett and Sandefur (2015) or Vivalt (2015) on external validity; Oster (2019) or Altonji et al. (2005) on…”
Evaluating solutions to the problem of false positives
Gall, T., & Maniadis, Z. (2018) · Research Policy · doi:10.1016/j.respol.2017.12.005 · Cited by 17
Bigoni, M., Camera, G., & Casari, M. (2019) · Journal of Monetary Economics · doi:10.1016/j.jmoneco.2019.01.002 · Cited by 20
Designing and Conducting Laboratory Experiments
Katok, E. (2018) · Wiley eBooks · doi:10.1002/9781119138341.ch1 · Cited by 29
Pownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z. M., Grinschgl, S., Elsherif, M. M., Gilligan-Lee, K. A., Oliveira, C. M., et al. (2022) · doi:10.31222/osf.io/9e526 · Cited by 16
Using the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22
Are the “Randomistas” Evaluators?
Picciotto, R. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0011 · Cited by 21
Trust-building vs. “just trust me”: reflexivity and resonance in ethnography
Pugh, A. J., & Mosseri, S. (2023) · Frontiers in Sociology · doi:10.3389/fsoc.2023.1069305 · Cited by 23
What meta‐research has taught us about research and changes to research practices
Ioannidis, J. P. A. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12666 · Cited by 25
Publication Biases in Replication Studies
Berinsky, A. J., Druckman, J. N., & Yamamoto, T. (2020) · Political Analysis · doi:10.1017/pan.2020.34 · Cited by 21
“Journals and institutions might encourage meta-replications projects such as the ones that have received some attention by aiming to replicate a large set of studies (e.g., Mullinix et al. 2015; Open Science Collaboration 2015; Camerer et al. 2016, 2018; Coppock 2019).”
BackgroundFrom Documents to Data: A Framework for Total Corpus Quality
Bodell, M. H., Magnusson, M., & Mützel, S. (2022) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231221135523 · Cited by 24
“…be more accustomed to the idea of result reproducibility (i.e., the potential that one researcher could take the original data and the code used for analysis in some prior study and reproduce all the numerical findings) (Camerer et al. 2016; Liu and Salganik 2019; Open Science Collaboration 2015).”
BackgroundThe logical structure of experiments lays the foundation for a theory of reproducibility
Buzbas, E. O., Devezer, B., & Baumgaertner, B. O. (2023) · Royal Society Open Science · doi:10.1098/rsos.221042 · Cited by 26
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 25
Silverstein, P., Pennington, C. R., Branney, P., O'Connor, D. B., Lawlor, E., O'Brien, E., & Lynott, D. (2024) · British Journal of Psychology · doi:10.1111/bjop.12700 · Cited by 27
“By now, evaluations of replicability have been conducted across a range of disciplines, from economics (Camerer et al., 2016) and experimental philosophy (Cova et al., 2021) to cardiovascular health (Prinz et al., 2011) and cancer biology (Begley & Ellis, 2012; Errington et al., 2021), reporting…”
Background
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.