Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 701–800 of 1,372
Produção Científica: um Guia Prático
Sampaio, M. I. C., Sabadini, A. A. Z. P., & Koller, S. H. (2022) · Universidade de São Paulo. Instituto de Psicologia eBooks · doi:10.11606/9786587596280 · Cited by 21
Dictator giving and taking: Evidence from Africa
Drouvelis, M. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.03.021 · Cited by 5
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Psychosomatic Medicine · doi:10.1097/psy.0000000000001186 · Cited by 5
Karabağ, S. F., Berggren, C., Pielaszkiewicz, J., & Gerdin, B. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-024-09520-z · Cited by 12
How Credible Is the Credibility Revolution?
Lang, K. (2024) · Journal of Labor Economics · doi:10.1086/732772 · Cited by 10
The Replication Database: Documenting the Replicability of Psychological Science
Röseler, L., Kaiser, L., Doetsch, C. A., Klett, N., Seida, C., Schütz, A., Aczél, B., Adelina, N., Agostini, V., Alarie, S., et al. (2024) · Journal of Open Psychology Data · doi:10.5334/jopd.101 · Cited by 28
Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Marcoci, A., Wilkinson, D. P., Vercammen, A., Wintle, B. C., Abatayo, A. L., Baskin, E., Berkman, H., Buchanan, E. M., Capitán, S., Capitán, T., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01961-1 · Cited by 21
“Replication probability was defined as the likelihood that independent researchers repeating the study would find a statistically significant effect in the same direction as the original study (that is, Q3 in Table SM6, Supplementary Information 6).”
InfluentialSelf-citationSealed Envelope Submissions Foster Research Integrity
Dufwenberg, M. A., & Martinsson, P. (2019) · Revue économique · doi:10.3917/reco.706.0919 · Cited by 5
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
Lehberger, M., & Grüner, S. (2021) · Food Policy · doi:10.1016/j.foodpol.2021.102100 · Cited by 11
Verhaltensökonomisch motivierte Maßnahmen zur Sicherung des Studienerfolgs (VStud)
Behlen, L., Brade, R., Himmler, O., & Jäckle, R. (2021) · Springer eBooks · doi:10.1007/978-3-658-32892-4_17 · Cited by 7
Cristea, I. A., Georgescu, R., & Ioannidis, J. P. A. (2021) · Clinical Psychological Science · doi:10.1177/21677026211049366 · Cited by 5
“For example, three large-scale studies have found that effects reported in multilaboratory preregistered replication attempts are on average 49% to 66% smaller than corresponding effects reported in previously published research (Camerer et al., 2016, 2018; Open Science Collaboration, 2015).”
ResultGerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
Competition and Prosociality: A Field Experiment in Ghana
Grosch, K., Ibáñez, M., & Viceisza, A. (2017) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2022.101887 · Cited by 15
Liu, T., Vickers, B. D., Seidler, R. D., & Preston, S. D. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1059051 · Cited by 8
de Gendre, A., Feld, J., & Salamanca, N. (2024) · Journal of Applied Econometrics · doi:10.1002/jae.3045 · Cited by 6
DISAPPOINTMENT AVERSION AND SOCIAL COMPARISONS IN A REAL‐EFFORT COMPETITION
Gächter, S., Huang, L., & Sefton, M. (2017) · Economic Inquiry · doi:10.1111/ecin.12498 · Cited by 8
Outlook and Next Steps: From Human Dynamics to Smart and Connected Communities
Sui, D. Z., & Shaw, S. (2018) · Human dynamics in smart cities · doi:10.1007/978-3-319-73247-3_13 · Cited by 4
Pot la psicologia rescatar-se a si mateixa?. Incentius, biaix i replicabilitat
Fernando, B., César, P. L. J., & Vadillo, M. A. (2018) · Anuari de Psicologia de la Societat Valenciana de Psicologia · doi:10.7203/anuari.psicologia.18.2.231 · Cited by 5
Editors’ Introduction: Controversies around RCT in Development
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0001 · Cited by 22
Mobley, M. L., Kruse, A. S., & McNickle, G. G. (2022) · Plant Direct · doi:10.1002/pld3.411 · Cited by 6
“As far as we can tell, plant biology has not been drawn into the replication crisis like other fields such as psychology (Pashler & Wagenmakers, 2012), medicine (Ioannidis, 2005), and economics (Camerer et al., 2016).”
BackgroundFecher, B., Frrrdorf, M., & Wagner, G. G. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2768629 · Cited by 6
“In a study that aimed to replicate 18 studies published in two top journals (American Economic Review and the Quarterly Journal of Economics) between 2011 and 2014, the researchers were able to find a significant effect in the same direction as the original study for 11 of their replications (61%) [17].”
BackgroundP-hacking in academic research
Ingre, M. (2016) · DiVA (Stockholm University) · doi:10.6084/m9.figshare.3393664 · Cited by 7
Robustness of the adult statistical word segmentation literature: Part 1Preprint
Hartshorne, J. K., Skorb, L., Dietz, S. L., Garcia, C., Iozzo, G., Lamirato, K., Ledoux, J., Mu, J., Murdock, K., Ravid, J., et al. (2017) · doi:10.31234/osf.io/vu8je · Cited by 5
Agricultural Productivity in Burkina Faso: The Role of Gender and Risk AttitudesPreprint
Sepahvand, M. H. (2022) · RePEc: Research Papers in Economics · Cited by 5
Colman, A. M., Gold, N., & Pulford, B. D. (2019) · Psychological Science · doi:10.1177/0956797619827880 · Cited by 5
“Experimental rigor relies on participants believing what they are told by experimenters, and the avoidance of deception in experimental economics has been cited as one of the principal reasons why economic experiments replicate more successfully than psychological experiments (Camerer et al., 2016).”
BackgroundReplication Markets: Results, Lessons, Challenges and Opportunities in AI ReplicationPreprint
Liu, Y., Gordon, M., Wang, J., Bishop, M., Chen, Y., Pfeiffer, T., Twardy, C. R., & Viganola, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04543 · Cited by 11
“The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, and other research fields (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Open Science Collaboration, 2015).”
Self-citationDo Seasoned Industry Specialists Provide Higher Audit Quality? A ResponsePreprint
Gaver, J. J., & Utke, S. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3745140 · Cited by 4
Post-Publication Peer Review for RealPreprint
Ikeda, K., Yamada, Y., & Takahashi, K. (2020) · doi:10.31234/osf.io/sp3j5 · Cited by 4
The "Tau" of Science - How to Measure, Study, and Integrate Quantitative and Qualitative KnowledgePreprint
Fanelli, D. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/67sak · Cited by 5
Assessing the replication landscape in experimental linguisticsPreprint
Kobrock, K., & Roettger, T. B. (2022) · doi:10.31234/osf.io/fzngs · Cited by 4
“…findings have uncovered alarmingly low rates of successful replications in fields such as psychology (Open Science Collaboration, 2015), economics (Camerer et al., 2016), and social sciences (Camerer et al., 2018), a state of affairs that has been referred to as the “replication crisis” (Fidler &…”
BackgroundAre Reference Points Merely Lagged Beliefs Over Probabilities?
Heffetz, O. (2018) · AEA Randomized Controlled Trials · doi:10.1257/rct.3061 · Cited by 7
Evaluating Research Transparency and Openness in Communication Sciences and Disorders JournalsPreprint
Schroeder, S. R., Gaeta, L. M., Amin, M. E., et al. (2022) · doi:10.31234/osf.io/dy5zs · Cited by 5
Bottlenecks for Evidence AdoptionPreprint
DellaVigna, S., Kim, W., & Linos, E. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4134940 · Cited by 5
Röseler, L., Gendlina, T., Krapp, J., Labusch, N., & Schütz, A. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/8psw2 · Cited by 7
Predicting the replicability of social and behavioural science claims in COVID-19 preprintsPreprint
Marcoci, A., Wilkinson, D. P., Abatayo, A. L., et al. (2023) · doi:10.31222/osf.io/xdsjf · Cited by 5
Self-citationImproving pension information: Experimental evidence on learning using online resources
Laroze, D., NOUSSAIR, C. N., Fajardo, G., Quintanilla, X., Zambrano, P. G., Vallette, P., & López-Tapia, M. (2025) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2025.101046 · Cited by 3
Pursuitworthiness in urgent research: Lessons on well-ordered science from sustainability science
Stojanovic, M. (2023) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2023.01.004 · Cited by 5
Schuurman, B., & Carthy, S. L. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292941 · Cited by 13
Сивак, Е., Pankowska, P., Mendrik, A. M., Emery, T., Garcia‐Bernardo, J., Hocuk, S., Karpińska, K., Maineri, A., Mulder, J., Nissim, M., et al. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00275-6 · Cited by 11
A protocol for structured robustness reproductions and replicability assessments
Peters, J., Brodeur, A., Dreber, A., Johannesson, M., Neubauer, F., & Rose, J. A. R. (2025) · Q Open · doi:10.1093/qopen/qoaf004 · Cited by 19
Self-citationThe Controversy behind the Controversies: Scientific Discourse in the Twenty-First Century
Fahnestock, J. (2025) · Rhetoric Society Quarterly · doi:10.1080/02773945.2025.2484162 · Cited by 6
Monéger, J., Chaby, L., Besche‐Richard, C., & Vergilino‐Perez, D. (2025) · Psychophysiology · doi:10.1111/psyp.70126 · Cited by 4
“…findings point to a low replicability rate in postural studies that parallels difficulties observed in other disciplines with low replication rates (Camerer et al. 2016; Ebersole et al. 2020; Errington et al. 2021; Evanschitzky and Armstrong 2010; Klein et al. 2014; Open Science Collaboration…”
Dreber, A., Johannesson, M., Nave, G., Apicella, C. L., Geniole, S. N., Imai, T., Knight, E. L., Manfredi, D., Mehta, P. H., Proietti, V. M., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2508519122 · Cited by 5
Self-citationHigh-profile journals put to reproducibility test
Ball, P. (2018) · Nature · doi:10.1038/d41586-018-06075-z · Cited by 4
Replication for Quantitative Research
Freese, J., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Nudging accurate scientific communication
Allard, A., & Clavien, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290225 · Cited by 6
“In psychology, only 36% of 97 targeted experiments were successfully replicated [1], compared to a slightly higher replicability rate of 61% in the related field of experimental economics [2].”
BackgroundEstimating the extent of selective reporting: An application to economics
Bruns, S. B., Deressa, T. K., Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1711 · Cited by 15
Truth, knowledge, and entrepreneurship theory: arguments for a rationalist scientific epistemology
Packard, M. D., & Bylund, P. L. (2025) · Small Business Economics · doi:10.1007/s11187-024-00993-1 · Cited by 8
“…swaths of empirical research in the social sciences, including economics and management, do not replicate (Aguinis et al., 2017; Bergh et al., 2017; Camerer et al., 2016a, 2016b; Open Science Collaboration, 2015; Świątkowski & Dompnier, 2017), a general consensus was quickly reached that we ought…”
The reproducibility opportunity
Macleod, M. R. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0398-0 · Cited by 4
Reproduction, replication, analysis and adaptation of a term alignment approach
Repar, A., Martinc, M., & Pollak, S. (2019) · Language Resources and Evaluation · doi:10.1007/s10579-019-09477-1 · Cited by 4
Internal replication of computational workflows in scientific research
Benjamin‐Chung, J., Colford, J. M., Mertens, A., Hubbard, A., & Arnold, B. F. (2020) · Gates Open Research · doi:10.12688/gatesopenres.13108.2 · Cited by 5
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Morduch, J. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0005 · Cited by 5
Replication to assess statistical adequacy
Owen, P. D. (2018) · Economics · doi:10.5018/economics-ejournal.ja.2018-60 · Cited by 4
“While there are differences in emphasis in replicating results from experimental data (Spanos, 2010; Camerer et al., 2016), testing for statistical adequacy also has a role to play in ensuring that the various aspects of experimental design have been successfully applied to generate data with the expected statistical…”
BackgroundThe endogenous formation of common pool resource coalitions
Chávez, C. A., Murphy, J. J., Quezada, F. J., & Stranlund, J. K. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.04.028 · Cited by 7
Do MTurkers exhibit myopic loss aversion?
Schwaiger, R., & Hueber, L. (2021) · Economics Letters · doi:10.1016/j.econlet.2021.110137 · Cited by 30
Lack of Replication or Generalization? Cultural Values Explain a Question Wording Effect
Silber, H., Tvinnereim, E. M., Stark, T., Blom, A. G., Krosnick, J. A., Bošnjak, M., Clement, S. L., Cornilleau, A., Cousteaux, A.-S., John, M., et al. (2021) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smab007 · Cited by 6
“During the last decade, the “crisis of replicability” has exploded across the sciences, including psychology (Pashler and Wagenmakers 2012), economics (Camerer et al. 2016), medicine (Ioannidis 2005), and other fields (Baker 2016).”
BackgroundWhite, M. A., & Burns, N. R. (2022) · Drug Science Policy and Law · doi:10.1177/20503245221119046 · Cited by 5
“…wrote his controversial paper on Why most published research findings are false, there have been a number of attempts to estimate the actual proportion of reported research findings that are false, and therefore do not replicate (e.g. Camerer et al., 2016, 2018; Open Science Collaboration, 2015).”
BackgroundCausality in Policy Studies: a Pluralist Toolbox
Damonte, A., & Negri, F. (2023) · Texts in quantitative political analysis · doi:10.1007/978-3-031-12982-7 · Cited by 4
Ibrahim, M., Chen, Y., Pereira, A. L., Musa, T., Bathia, D., & Handa, S. (2023) · Journal of African Economies · doi:10.1093/jae/ejad013 · Cited by 4
Thibault, R. T., Kovács, M., Hardwicke, T. E., Sarafoglou, A., Ioannidis, J. P. A., & Munafo, M. R. (2023) · Royal Society Open Science · doi:10.1098/rsos.230568 · Cited by 6
Quantifying replicability of multiple studies in a meta-analysis
Xiao, M., Chu, H., Hodges, J. S., & Lin, L. (2024) · The Annals of Applied Statistics · doi:10.1214/23-aoas1806 · Cited by 5
Representative methods of computational socioeconomics
Zhou, T. (2021) · Journal of Physics Complexity · doi:10.1088/2632-072x/ac2072 · Cited by 10
Factor Investing in Sovereign Bond Markets: Deep Sample Evidence
Baltussen, G., Martens, M. P. E., & Penninga, O. (2021) · The Journal of Portfolio Management · doi:10.3905/jpm.2021.1.311 · Cited by 7
Moore, D. A., Thau, S., Zhong, C., & Gino, F. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.104111 · Cited by 4
First principles, fallibilism, and economics
Hoover, K. D. (2018) · Synthese · doi:10.1007/s11229-018-02021-8 · Cited by 4
‘I’m Just a Soul Whose Intentions Are Good’: The Role of Communication in Noisy Repeated GamesPreprint
Arechar, A. A., Dreber, A., Fudenberg, D., & Rand, D. G. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2748890 · Cited by 4
Self-citationSocial Identity and Group ContestsPreprint
Zaunbrecher, H. W., & Riedl, A. M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2816038 · Cited by 4
“Camerer et al. (2016) fail to replicate Chen and Chen (2011) and Guala and Filippin (2015) show widely inconsistent effects of social identity across different contexts in modified dictator games.”
Herz, H., & Taubinsky, D. (2016) · National Bureau of Economic Research · doi:10.3386/w22728 · Cited by 3
Arguing Practical Significance in Software Engineering Using Bayesian Data Analysis.Preprint
Torkar, R., FELDT, R. H., & Furia, C. A. (2018) · arXiv (Cornell University) · Cited by 3
Truth, Proof, and Reproducibility: There's no counter-attack for the codelessPreprint
Gray, C. T., & Marwick, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05947 · Cited by 4
“How we re-produce scientific results is being examined across a range of disciplines [6,35].”
BackgroundMcCabe, M. J., & Mueller‐Langer, F. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3329272 · Cited by 5
Reflecting on the Evidence: A Reply to Knight, McShane, et al. (2020)
Nave, G., Daviet, R., Nadler, A., Zava, D. T., & Camerer, C. F. (2020) · Psychological Science · doi:10.1177/0956797620930966 · Cited by 3
“’s work is via a well-powered, direct replication (Brandt et al., 2014; Camerer et al., 2016; Camerer et al., 2018; Knight, Sarkar, Prasad, & Mehta, 2020; Simons, 2014) We are currently collaborating with some of Knight, McShane, et al.”
MethodSelf-citationMuradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2020) · doi:10.31222/osf.io/wvdjf · Cited by 4
I enjoy hurting my classmates: On the relation of boredom and sadism in schoolsPreprint
Pfattheicher, S., Lazarević, L. B., Nielsen, Y. A., et al. (2021) · doi:10.31234/osf.io/37ru9 · Cited by 4
Is Open Science Neoliberal?Preprint
Tunç, D. U., Tunç, M. N., & Eper, Z. B. (2021) · doi:10.31234/osf.io/ft8dc · Cited by 4
Osborne, C., & Norris, E. (2022) · doi:10.31234/osf.io/w3qg9 · Cited by 4
Scales, D. A., Gorman, J. M., Leff, C., & Gorman, S. E. (2021) · MediArXiv (OSF Preprints) · doi:10.33767/osf.io/rcugs · Cited by 7
Eitan, O., Viganola, D., Inbar, Y., et al. (2018) · doi:10.31234/osf.io/2vuhb · Cited by 4
Self-citationFoundations and Futures of Strategic ManagementPreprint
Leiblein, M. J., & Reuer, J. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3396754 · Cited by 4
Bayesian evaluation of effect size after replicating an original studyPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/g3sne · Cited by 5
Friendship and partner choice in rural ColombiaPreprint
Redhead, D., Ross, C. T., & Ragione, A. D. (2022) · doi:10.31234/osf.io/ekwu8 · Cited by 4
Heterogeneity in effect size estimates: Empirical evidence and practical implicationsPreprint
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · doi:10.31222/osf.io/583un · Cited by 4
Self-citationThe Economics of Computational ReproducibilityPreprint
Hurlin, C., Colliard, J.-É., & Pérignon, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4811513 · Cited by 6
ReplicationSuccess: Design and Analysis of Replication Studies
Held, L., Micheloud, C., Gerber, F., & Hofmann, F. (2021) · doi:10.32614/cran.package.replicationsuccess · Cited by 4
Coville, A., & Vivalt, E. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5nsh3_v1 · Cited by 19
Fellnhofer, K., Wennberg, K. J., Allison, T. H., Arenius, P. M., Lévesque, M., Gish, J. J., & Pollack, J. M. (2026) · Entrepreneurship Theory and Practice · doi:10.1177/10422587251401175 · Cited by 6
Research into evidence-based psychological interventions needs a stronger focus on replicability
Niemeyer, H., Knaevelsrud, C., van Aert, R. C. M., & Ehring, T. (2023) · Clinical Psychology in Europe · doi:10.32872/cpe.9997 · Cited by 3
“…on this issue, in the areas of psychology (Klein et al., 2014; Klein et al., 2018; Open Science Collaboration [OSC], 2015), economics (e.g., Camerer et al., 2016), epidemiology (e.g., Kaltiala-Heino, Työläjärvi, & Lindberg, 2019; Zisook et al., 2007) and medicine (Errington, Denis,…”
Evaluating meta-analysis as a replication success measure
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308495 · Cited by 9
DeScAI: the convergence of decentralized science and artificial intelligence
Shilina, S. (2025) · Frontiers in Blockchain · doi:10.3389/fbloc.2025.1657050 · Cited by 5
Predicting replication outcomes and generalizability of results
Dreber, A. (2025) · European Review of Agricultural Economics · doi:10.1093/erae/jbaf035 · Cited by 3
Self-citationThe Systematic Multiverse Analysis Registration Tool for defining multiverse analyses
Short, C. A., Inceler, Y. C., Frank, M., & Hildebrandt, A. (2025) · Royal Society Open Science · doi:10.1098/rsos.250800 · Cited by 5
Hamilton, D. G., McKenzie, J. E., Nguyen, P., Rethlefsen, M. L., McDonald, S. J., Brennan, S. E., Fidler, F., Higgins, J. P. T., Kanukula, R., Karunananthan, S., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2025.10064 · Cited by 8
Chandrashekar, S. P., Viganola, D., Dreber, A., Johannesson, M., Pfeiffer, T., Siegel, A., & Feldman, G. (2026) · Royal Society Open Science · doi:10.1098/rsos.250377 · Cited by 5
Self-citationTeaching Reproducible Methods in Economics at Liberal Arts Colleges: A Survey
Underwood, A., Sichel, A., & Marshall, E. C. (2023) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2023.2234425 · Cited by 4
“In economics, researchers have found that less than half to two-thirds of papers studied are replicable (Camerer et al. 2016; Chang and Li 2017).”
Blanchard, R. C., DeWitt, T. J., Young, S., & Perkin, J. S. (2024) · Journal of Fish Biology · doi:10.1111/jfb.15665 · Cited by 3
Lindkvist, A. M., Koppel, L., & Tinghög, G. (2024) · Scientific Reports · doi:10.1038/s41598-024-53450-0 · Cited by 8
Neural Diversity and Decisions
Thoelen, G., & Zak, P. J. (2024) · Adaptive Human Behavior and Physiology · doi:10.1007/s40750-024-00237-2 · Cited by 3
“Due in part to this approach, a swath of the published findings have failed to replicate (Camerer et al., 2016, 2018; Nosek et al., 2022).”
Fairness preferences revisited
Zhang, Y., Hoffmann, M., Sara, R., & Eckel, C. C. (2024) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.04.033 · Cited by 3
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.