Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationVariability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
“18 (9), Brazil (3), Canada (16), China (2), Finland (1), France (4), Germany (21), Italy (13), the Netherlands (8), Spain (4), Sweden (4), Switzerland (3), Taiwan (3), UK (7) and the USA (76).”
Self-citationHou, K., Xue, C., & Zhang, L. (2020) · The Review of Financial Studies · doi:10.1093/rfs/hhy131 · Cited by 989
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationMuthukrishna, M., & Henrich, J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0522-1 · Cited by 638
A unified welfare analysis of government policies.
Hendren, N., & Sprung-Keyser, B. (2020) · The Quarterly Journal of Economics · doi:10.1093/qje/qjaa006 · Cited by 571
Small Effects: The Indispensable Foundation for a Cumulative Psychological Science
Götz, F. M., Gosling, S. D., & Rentfrow, P. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620984483 · Cited by 538
“In doing so, the publication culture will contribute to the lack of replicability plaguing the social sciences in general (Camerer et al., 2016, 2018) and psychology in particular (Open Science Collaboration, 2015).”
BackgroundThe past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 480
Financial Education Affects Financial Knowledge and Downstream Behaviors
Kaiser, T., Lusardi, A., Menkhoff, L., & Urban, C. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2021.09.022 · Cited by 508
Soto, C. J. (2019) · Psychological Science · doi:10.1177/0956797619831612 · Cited by 470
“We therefore adopted an approach that attempted to replicate a large number of original effects from many studies, with one replication attempt per effect and relatively brief outcome measures (Camerer et al., 2016; Cova et al., 2018; Open Science Collaboration, 2015).”
BackgroundMethodResultInfluentialMultiple hypothesis testing in experimental economics
List, J. A., Shaikh, A. M., & Xu, Y. (2019) · Experimental Economics · doi:10.1007/s10683-018-09597-5 · Cited by 463
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
“Interestingly too, economics experiments (Camerer et al., 2016) have a much higher replicability rate than do psychology experiments (Open Science Collaboration, 2015); again, the reason for this finding is not clear but it may have an explanation premium on causal identification, (c) incentivize…”
BackgroundTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
“One of the most important recent studies is Camerer et al. (2016), which repeated 18 behavioral economics lab experiments originally published between 2011 and 2014 in the American Economic Review and the Quarterly Journal of Economics to assess their replicability.”
InfluentialZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. (2018) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 392
“The NLS model, as (12) stresses, matches the expected log effort, while the minimum-distance matches the log of expected effort (given the assumed homogeneity).”
InfluentialIdentification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Proxies and Databases in Financial Misconduct Research
Karpoff, J. M., & Koester, A. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2811778 · Cited by 329
“similar to, or lower than, the replication rates recently reported in such other sciences as psychology, biomedicine, and experimental economics (see Open Science Collaboration 2015; Iqbal, Wallach, Khoury, Schully, and Ioannidis 2016; Camerer et al. 2016).”
Ten considerations for effectively managing the COVID-19 transition
Habersaat, K. B., Betsch, C., Danchin, M., Sunstein, C. R., Böhm, R., Falk, A., Brewer, N. T., Omer, S. B., Scherzer, M., Sah, S., et al. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0906-x · Cited by 297
Outcome-Wide Longitudinal Designs for Causal Inference: A New Template for Empirical Studies
VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2020) · Statistical Science · doi:10.1214/19-sts728 · Cited by 310
“for the recent so-called “replication crisis” (Open Science Collaboration, 2015; Camerer et al., 2016).”
The Behavioral Revolution and International Relations
Hafner-Burton, E. M., Haggard, S., Lake, D. A., & Victor, D. G. (2017) · International Organization · doi:10.1017/s0020818316000400 · Cited by 291
“Camerer et al. 2016; Open Science Collaboration 2015.”
BackgroundSwedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Testing the Waters: Behavior across Participant Pools
Snowberg, E., & Yariv, L. (2021) · American Economic Review · doi:10.1257/aer.20181065 · Cited by 273
“In the parlance of the literature concerned with the reliability of social science findings: the result we find in the CCS sample does not replicate in the MTurk and Representative samples (Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundResultPublication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
“In the development of a scientific field, certain claims stand out as both significant and stable in the face of further experimentation [1].”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
“, 2011), efforts by a nonprofit biotech to replicate findings of potential drugs in a mouse model of amyotrophic lateral sclerosis (Perrin, 2014), and systematic replication efforts in other disciplines (Camerer et al., 2016; Camerer et al., 2018; Cova et al., 2018; Ebersole et al., 2016; Ebersole et al., 2019; Klein…”
BackgroundComparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Self-citationThe readability of scientific texts is decreasing over time
Plavén-Sigray, P., Matheson, G. J., Schiffler, B. C., & Thompson, W. H. (2017) · eLife · doi:10.7554/elife.27725 · Cited by 257
Three Recommendations for Improving the Use of p -Values
Benjamin, D. J., & Berger, J. O. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543135 · Cited by 255
“We believe that this statistical fact has contributed to the alarmingly high levels of nonreplicability that have been documented in several research communities (e.g., Open Science Collaboration 2015; Camerer et al. 2016; Johnson et al. 2017).”
BackgroundRafi, Z., & Greenland, S. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01105-9 · Cited by 243
“Webelieve thatneglect of theseproblems is amajor contributor to the current crisis of statistics in science [4–9].”
Redefine statistical significance
Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K., Brembs, B., Brown, L., Camerer, C., et al. (2017) · doi:10.31234/osf.io/mky9j · Cited by 211
Self-citationAn Agenda for Open Science in Communication
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito, J., Bier, L. M., Zhang, R., Johnson, B. K., et al. (2021) · Journal of Communication · doi:10.1093/joc/jqz052 · Cited by 209
P. Simmons, J., D. Nelson, L., & Simonsohn, U. (2021) · Journal of Consumer Psychology · doi:10.1002/jcpy.1208 · Cited by 213
Musical Preferences Predict Personality: Evidence From Active Listening and Facebook Likes
Nave, G., Minxha, J., Greenberg, D. M., Kosinski, M., Stillwell, D., & Rentfrow, J. (2018) · Psychological Science · doi:10.1177/0956797618761659 · Cited by 206
“Extensive research and discussions devoted to estimating the replicability of laboratory experiments in psychology, and social sciences in general, have highlighted the critical importance of replication efforts (Camerer et al., 2016; Carter & McCullough, 2014; Lane, Luminet, Nave, & Mikolajczak, 2016; Nave, Camerer,…”
BackgroundSelf-citationHardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
RCTs to Scale: Comprehensive Evidence from Two Nudge Units
DellaVigna, S., & Linos, E. (2020) · National Bureau of Economic Research · doi:10.3386/w27594 · Cited by 189
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. (2018) · Journal of Political Economy · doi:10.1086/699976 · Cited by 183
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
Nonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177
Aguinis, H., Hill, N. S., & Bailey, J. R. (2021) · Organizational Research Methods · doi:10.1177/1094428119836485 · Cited by 180
“But, this is not unique to organizational research, and the same challenge is currently faced by economics (Camerer et al., 2016) and the natural sciences as well (Baker, 2016).”
BackgroundThe Prevalence of Marginally Significant Results in Psychology Over Time
Olsson-Collentine, A., van Assen, M. A. L. M., & Hartgerink, C. H. J. (2019) · Psychological Science · doi:10.1177/0956797619830326 · Cited by 178
“is important as it bears on differences in reproducibility across disciplines and trends over time; higher p values are generally associated with lower reproducibility and more false positives (Camerer et al., 2016; Ioannidis, 2005; Open Science Collaboration, 2015).”
BackgroundThe Generalizability of Online Experiments Conducted During the COVID-19 Pandemic
Peyton, K., Huber, G. A., & Coppock, A. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2021.17 · Cited by 174
“In economics, Camerer et al. (2016) replicated 18 survey and lab-based experiments and found effect sizes in the 11 studies that successfully replicated were approximately 66% of the original.”
BackgroundCrowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
“…work has demonstrated that researchers can anticipate whether a published result will independently replicate based on the research report alone (Camerer et al., 2016; Dreber et al., 2015), and predict the effects of performance interventions starting only from a few benchmark effects and the…”
Self-citationLeuz, C. (2018) · Accounting and Business Research · doi:10.1080/00014788.2018.1470151 · Cited by 181
Enhancing Rigor in Quantitative Entrepreneurship Research
Maula, M., & Stam, W. (2020) · Entrepreneurship Theory and Practice · doi:10.1177/1042258719891388 · Cited by 167
“fallibility of research findings (e.g., Baker, 2016; Camerer et al., 2016; Gelman, 2018; Honig et al., 2018; Miguel et al., 2014; Open Science Collaboration, 2015), insufficient replication and replicability (Bettis et al., 2016), and the growing maturity of the field.”
BackgroundAn umbrella review of randomized control trials on the effects of physical exercise on cognition
Ciria, L. F., Román-Caballero, R., Vadillo, M. A., Holgado, D., Luque-Casado, A., Perakakis, P., & Sanabria, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01554-4 · Cited by 157
Replicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 190
“Replication projects in medicine (Prinz et al., 2011) and behavioral economics (Camerer et al., 2016) resulted in replication rates of 25 and 61%, respectively, and analyses in genetics (Munafò, 2009) and neuroscience (Button et al., 2013) question the validity of studies in those fields.”
BackgroundStrategic Intelligence: The Cognitive Capability to Anticipate Competitor Behavior
Levine, S. S., Bernard, M. M., & Nagel, R. (2017) · Strategic Management Journal · doi:10.1002/smj.2660 · Cited by 186
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Editorial: Enhancing quantitative theory-testing entrepreneurship research
Anderson, B. S., Wennberg, K. J., & McMullen, J. S. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2019.02.001 · Cited by 184
Forecasting in social settings: The state of the art
Makridakis, S., Hyndman, R. J., & Petropoulos, F. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.05.011 · Cited by 190
The replicability crisis and public trust in psychological science
Anvari, F., & Lakens, D. (2018) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2019.1684822 · Cited by 176
“%) cases (Camerer et al., 2016).”
Philosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 184
Statistical Nonsignificance in Empirical Economics
Abadie, A. (2020) · American Economic Review Insights · doi:10.1257/aeri.20190252 · Cited by 207
Predict science to improve science
DellaVigna, S., Pope, D., & Vivalt, E. (2019) · Science · doi:10.1126/science.aaz1704 · Cited by 136
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 171
Validation of cluster analysis results on validation data: A systematic framework
Ullmann, T., Hennig, C. M., & Boulesteix, A. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1444 · Cited by 177
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117
Self-citationThe Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Bryant, B. R., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 143
“Interestingly, when researchers have conducted replication studies, results have often failed to validate original findings (Camerer et al., 2016; Ebersole et al., 2016; Klein et al., 2014; Open Science Collaboration, 2015).”
ResultValue-based attentional capture affects multi-alternative decision making
Gluth, S., Spektor, M. S., & Rieskamp, J. (2018) · eLife · doi:10.7554/elife.39659 · Cited by 137
The influence of hidden researcher decisions in applied microeconomics
Huntington‐Klein, N., Arenas, A., Beam, E., Bertoni, M., Bloem, J. R., Burli, P., Chen, N., Grieco, P. L. E., Ekpe, G. K., Pugatch, T., et al. (2021) · Economic Inquiry · doi:10.1111/ecin.12992 · Cited by 190
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 144
Conceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 124
LIONESS Lab – A Free Web-Based Platform for Conducting Interactive Experiments Online
Giamattei, M., Yahosseini, K. S., Gächter, S., & Molleman, L. (2019) · Journal of the Economic Science Association · doi:10.1007/s40881-020-00087-0 · Cited by 158
Replication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 145
“Replicability has been described as an ideal standard of good scientific practice (Popper, 1959; Jasny et al., 2011; Campbell and Stanley, 1963).”
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22566 · Cited by 124
What Is Meant by “Replication” and Why Does It Encounter Resistance in Economics?
Duvendack, M., Palmer‐Jones, R., & Reed, W. R. (2017) · American Economic Review · doi:10.1257/aer.p20171031 · Cited by 147
“While similar to Open Science Collaboration (2015) in many respects, they report a higher replication rate.6 Chang and Li (2015) and Duvendack, Palmer-Jones, and Reed (2015) both measure replication rates in economics, with both reporting relatively low rates of replication success.”
ResultTransparency of CHI Research Artifacts: Results of a Self-Reported Survey
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376448 · Cited by 118
“INTRODUCTION Recently, scientific disciplines such as psychology [9], medicine [3], economics [6], and political science [16] have had many published research articles fail to replicate, i.”
BackgroundHow to run an experimental auction: a review of recent advances
Canavari, M., Drichoutis, A. C., Lusk, J. L., & Nayga, R. M. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz038 · Cited by 140
“Given the well known general lack of reproducibility of scientific studies in economics (Camerer et al., 2016) and psychology (Open Science Collaboration, 2015) respectively, there is growing concern over the credibility of claims of new discoveries based on statistically significant findings.(9) There have been sever…”
BackgroundInfluentialCommon methodological mistakes
Wulff, J. N., Sajons, G. B., Pogrebna, G., Lonati, S., Bastardoz, N., Banks, G. C., & Antonakis, J. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101677 · Cited by 161
Data, measurement and empirical methods in the science of science
Liu, L., Jones, B. F., Uzzi, B., & Wang, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01562-4 · Cited by 180
Wang, S. V., Sreedhara, S. K., Schneeweiß, S., Initiative, R., Franklin, J. M., Gagne, J. J., Huybrechts, K. F., Patorno, E., Jin, Y., Lee, M., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-32310-3 · Cited by 139
Vadillo, M. A., Gold, N., & Osman, M. (2016) · Psychological Science · doi:10.1177/0956797616654911 · Cited by 141
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
“(2014), 36 of 97 findings (37%) in the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015)1, 11 of 18 findings (61%) in Camerer et al. (2016), 3 of 10 findings (30%) in Ebersole et al. (2016), 29 of 37 findings (78%) in Cova et al. (2018), 13 of 21 findings (62%) in Camerer et al.”
BackgroundResultInfluentialSelf-citationBest Practice Recommendations for Replicating Experiments in Public Administration
Walker, R. M., Brewer, G. A., Lee, M. J., Petrovsky, N., & van Witteloostuijn, A. C. (2018) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muy047 · Cited by 117
Failures in Contingent Reasoning: The Role of Uncertainty
Martínez-Marquina, A., Niederle, M., & Vespa, E. (2019) · American Economic Review · doi:10.1257/aer.20171764 · Cited by 146
“Stahl and Wilson (1995), Camerer et al. (2004), Eyster and Rabin (2005), Jehiel (2005), Crawford and Iriberri (2007), Esponda (2008), Crawford et al.”
BackgroundInfluentialWhat do we really know about the acceptance of battery electric vehicles? – Turns out, not much
Wicki, M., Brückmann, G., Quoß, F., & Bernauer, T. (2022) · Transport Reviews · doi:10.1080/01441647.2021.2023693 · Cited by 125
“Given the replication crisis in various areas of the sciences (Camerer et al., 2016; Fanelli, 2010), we firmly believe that both scientists and policy-makers should have more confidence in findings that third parties can replicate,”
BackgroundMENKVELD, A. J., DREBER, A., HOLZMEISTER, F., HUBER, J., JOHANNESSON, M., KIRCHLER, M., NEUSÜß, S., RAZEN, M., WEITZEL, U., ABAD‐DÍAZ, D., et al. (2024) · The Journal of Finance · doi:10.1111/jofi.13337 · Cited by 113
Self-citationCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“…is that effect sizes observed in the replication studies have been on average approximately half as large as those reported in the original studies (Camerer et al. 2016, 2018), consistent with the idea that most published effects are inflated by selective reporting and other biases (Ioannidis…”
Open and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 126
Estimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 119
“10 EERP (7) 18 Economics 2 2011 to 2014 Experiments Single-lab, Published Test set III same method 11 ERW (28) 122 Economics 45 1973 to 2015 Experiments, correlational 1) Same data, Published Test set IV studies, & modeling same code 2) New data, same methods 3) Same data, new methods 4) New methods,”
BackgroundMethodThe Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · Southern Economic Journal · doi:10.1002/soej.12392 · Cited by 116
“39 Abadie (2018) provides a numerical example for constructing a prior distribution for experimental economics studies, using estimates from a replication project we discuss in Section 5 (Camerer et al., 2016; Andrews and Kasy, 2017).”
InfluentialPredicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 111
Self-citationPredicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225826 · Cited by 118
“The data are combined from four replication projects, The Reproducibility Project in Psychology (RPP; [7]), the Experimental Economics Replication Project (EERP; [2]) and Many Labs (ML) 1 and 3 [28, 29].”
BackgroundMethodInfluentialSelf-citationÖzer, Ö., Subramanian, U., & Wang, Y. (2017) · Management Science · doi:10.1287/mnsc.2016.2617 · Cited by 101
Reproducibility and Replicability in Geographical Analysis
Kedron, P., Frazier, A. E., Trgovac, A. B., Nelson, T., & Fotheringham, A. S. (2021) · Geographical Analysis · doi:10.1111/gean.12221 · Cited by 94
Replications in Comparative Cognition: What Should We Expect and How Can We Improve?
Farrar, B. G., Boeckle, M., & Clayton, N. S. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.01.02.2020 · Cited by 97
“In psychology and related fields, large-scale attempts to replicate published research have only produced significant results around 60% of the time (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2018; Open Science Collaboration, 2015; Zwaan et al., 2018).”
The globalizability of temporal discounting
Ruggeri, K., Panin, A., Vdović, M., Većkalov, B., Abdul-Salaam, N., Achterberg, J., Akil, C., Amatya, J., Amatya, K., Andersen, T. L., et al. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01392-w · Cited by 130
“For good reason, psychological theory has come under considerable recent criticism due to a number of failed replications of previously canonical constructs 51 .”
Justify Your Alpha: A Primer on Two Practical Approaches
Maier, M., & Lakens, D. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221080396 · Cited by 155
“Many empirical sciences have recently been troubled by a replication crisis (Camerer et al., 2016; Open Science Collaboration, 2015), which has in part been caused by inflated alpha levels because of p -hacking (Simmons et al., 2011), publication bias, and low statistical power (Lindsay, 2015).”
Effort, luck, and voting for redistribution
Lefgren, L. J., Sims, D., & Stoddard, O. B. (2016) · Journal of Public Economics · doi:10.1016/j.jpubeco.2016.08.012 · Cited by 121
“Recent research suggests that one-third of laboratory experiments published in recent years in the top general interest journals in Economics are not replicable in new samples (Camerer et al., 2016).”
BackgroundSoto, C. J. (2021) · Social Psychological and Personality Science · doi:10.1177/1948550619900572 · Cited by 90
“Moreover, despite recent metascientific scholarship questioning the replicability of behavioral science (Camerer et al., 2016, 2018; Cova et al., in press; Open Science Collaboration, 2015; Simmons et al., 2011), many links between personality traits and life outcomes can be replicated (Soto, 2019).”
BackgroundPollet, T. V., & Saxton, T. K. (2019) · Evolutionary Psychological Science · doi:10.1007/s40806-019-00192-2 · Cited by 89
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.