p-Curve and p-Hacking in Observational Research
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
In the citing paper
“Methodological criticisms suggest that these and similar studies may be underestimating the true impact of P -hacking (11, 12).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [11]
In the citing paper
“P-hacking22,23 (where investigators perform multiple analyses sequentially, for example by adding more samples or excluding some potential outliers until the desired thresholds of statistical significance are achieved)”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWitten, I. H. (2008) · Cited by 9,143
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“The p -curve was also used to infer that most studies actually analyze true effects challenging previous claims [36].”
MethodSelf-citationGreene, W. (2000) · University of Maribor digital library (University of Maribor) · Cited by 3,746
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 3×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
BackgroundLet's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach
Sala-i-Martin, X., Doppelhofer, G., & Miller, R. I. (2004) · American Economic Review · doi:10.1257/0002828042002570 · Cited by 2,112
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
Background
Cited by
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579
Heisig, J. P., & Schaeffer, M. (2019) · European Sociological Review · doi:10.1093/esr/jcy053 · Cited by 552
“Second, and more importantly, a narrow interpretation of p-curve analysis has come under attack by Bruns and Ioannidis (2016), who argue that in observational studies omitted variable biases may create right skewed p-curves even in the absence of an underlying causal effect.”
BackgroundThe past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
The ABCD study: understanding the development of risk for mental and physical health outcomes
Karcher, N. R., & Barch, D. M. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0736-6 · Cited by 351
“Further, preregistering analysis approaches even in the absence of specific hypotheses helps to avoid problematic analytic approaches resulting in enhanced false positives [73, 74].”
BackgroundWhat meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
“This is due to how NHST tests statistics are computed, the properties of real data and to the lack of specifying data predicted by H1 (Bruns and Ioannidis, 2016).”
BackgroundInfluentialSelf-citationReporting guidelines for meta-analysis in economics
Havránek, T., Stanley, T. D., Doucouliagos, H., Bom, P., Geyer‐Klingeberg, J., Iwasaki, I., Reed, W. R., Rost, K., & van Aert, R. C. M. (2020) · Journal of Economic Surveys · doi:10.1111/joes.12363 · Cited by 271
Thoral, P. J., Peppink, J. M., Driessen, R. H., Sijbrands, E. J. G., Kompanje, E. J. O., Kaplan, L., Bailey, H., Kesecioglu, J., Cecconi, M., Churpek, M., et al. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004916 · Cited by 257
“In addition, it may aid in addressing the problem of irreproducibility in research as analyses can easily be replicated, especially if code-sharing is enforced (5, 6).”
BackgroundLimitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229
“…approach is not valid (biased and unreliable) when there is heterogeneity, misspecification biases, or when some insignificant studies are reported (Bruns & Ioannidis, 2016; McShane, Böckenholt, & Hansen, 2016; van Aert, Wicherts, & van Assen, 2016), conditions ubiquitous in the social sciences.”
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
“Ulrich and Miller (2015) and Bruns and Ioannidis (2016) illustrated that multiple p-hacking behaviors may result in relatively more small p values, which leads to overestimation of effect size in p-curve (and p-uniform). We examined the effect of two methods of p-hacking on effect-size estimation in p-curve and p-unif…”
BackgroundMethodThreats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 155
“7 review articles its first 10 years, assisted by the decision of the International Committee of Medical Journal Editors to make pre-registration a requirement for publication in their journals.”
BackgroundTransparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201
Big little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 208
“In the past, several methods have been applied for p-hacking detection, such as p-curve [12,34], the Caliper test [61,103], Fisher’s method [104] or excess test statistics [13].”
BackgroundMethodIoannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175
Self-citationMeasuring capital-labor substitution: The importance of method choices and publication bias
Gechert, S., Havránek, T., Iršová, Z., & Kolcunová, D. (2021) · Review of Economic Dynamics · doi:10.1016/j.red.2021.05.003 · Cited by 171
“…related to quality (such 2Publication bias in economics has also been recently discussed, among others, by Havranek (2015), Brodeur et al. (2016), Bruns & Ioannidis (2016), Havranek & Irsova (2017), Havranek et al. (2017), Christensen & Miguel (2018), Astakhov et al. (2019), Bajzik et al.…”
BackgroundDesigning studies of predation risk for improved inference in carnivore-ungulate systems
Prugh, L. R., Sivy, K. J., Mahoney, P. J., Ganz, T. R., Ditmer, M. A., van de Kerk, M., Gilbert, S. L., & Montgomery, R. A. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.02.011 · Cited by 165
“The “big data” which these technologies provide must be analyzed carefully to avoid overstating statistical power (Bruns and Ioannidis, 2016).”
BackgroundBest practices in developing, conducting, and evaluating inductive research
Woo, S. E., O'Boyle, E. H., & Spector, P. E. (2017) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.08.004 · Cited by 160
What Have We (Not) Learnt from Millions of Scientific Papers with P Values?
Ioannidis, J. P. A. (2019) · The American Statistician · doi:10.1080/00031305.2018.1447512 · Cited by 159
“Such a bias, through tiny, can generate a spurious P-curve that resembles genuine discoveries when in fact it is a mere artifact of confounding and omitted variable bias (Bruns and Ioannidis 2016).”
BackgroundSelf-citationEstimating the Armington elasticity: The importance of study design and publication bias
Bajzik, J., Havranek, T., Irsova, Z., & Schwarz, J. (2020) · Journal of International Economics · doi:10.1016/j.jinteco.2020.103383 · Cited by 150
Patterns of MHC ‐dependent mate selection in humans and nonhuman primates: a meta‐analysis
Winternitz, J. C., Abbate, J. L., Huchard, É., Havlíček, J., & Garamszegi, L. Z. (2016) · Molecular Ecology · doi:10.1111/mec.13920 · Cited by 141
Vadillo, M. A., Gold, N., & Osman, M. (2016) · Psychological Science · doi:10.1177/0956797616654911 · Cited by 105
“It is worth noting that, as with any other statistical test, p-curve analysis is not a flawless indicator of bias (Bishop & Thompson, 2016; Bruns & Ioannidis, 2016; Lakens, 2015).”
BackgroundA kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
Gómez-de-Mariscal, E., Guerrero, V., Sneider, A., Jayatilaka, H., Phillip, J. M., Wirtz, D., & Muñoz-Barrutia, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-00199-5 · Cited by 104
“(9), where θα,γ = 1 means that there are practical differences among the compared samples.”
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Elliott, G., Kudrin, N., & Wüthrich, K. (2022) · Econometrica · doi:10.3982/ecta18583 · Cited by 113
“Instead of the true distribution of p-values, g(p), (4)We refer to Bruns and Ioannidis (2016) and Ulrich and Miller (2015) for alternative examples of p-hacking that lead to non-increasing p-curves.”
BackgroundMethodLaraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 103
“Some of these QRPs fall under the term “p-hacking,” which can include “choosing to report a subset of multiple independent variables or adding observations [data points] until the effect of interest is significant” (Bruns & Ioannidis, 2016).”
BackgroundPavlacic, J. M., Buchanan, E. M., Maxwell, N. P., Hopke, T. G., & Schulenberg, S. E. (2019) · Review of General Psychology · doi:10.1177/1089268019831645 · Cited by 87
Nitschke, F. T., McKimmie, B. M., & Vanman, E. J. (2019) · Psychological Bulletin · doi:10.1037/bul0000206 · Cited by 55
Who watches the watchmen? Evaluating evaluations of El Sistema
Baker, G., Bull, A., & Taylor, M. (2018) · British Journal of Music Education · doi:10.1017/s0265051718000086 · Cited by 80
“This is a technique known as p-hacking (see e.g. Head et al., 2015; Bruns & Ioannidis, 2016).”
MethodPublication and Attenuation Biases in Measuring Skill Substitution
Havránek, T., Iršová, Z., Laslopová, Ľ., & Zeynalova, O. (2024) · The Review of Economics and Statistics · doi:10.1162/rest_a_01227 · Cited by 75
Gülöksüz, S., Rutten, B. P. F., Pries, L.-K., ten Have, M., de Graaf, R., van Dorsselaer, S., Klingenberg, B., van Os, J., Ioannidis, J. P. A., & The European Network of National Schizophrenia Networks Studying Gene-Environment Interactions Work Package 6 (EU-GEI WP6) Group (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby118 · Cited by 78
Self-citationResearch Misconduct in Business and Management Studies: Causes, Consequences, and Possible Remedies
Tourish, D., & Craig, R. (2018) · Journal of Management Inquiry · doi:10.1177/1056492618792621 · Cited by 73
“…that deliver a desired p value, terminating a study when a desired p value has been reached, dropping items from survey instruments that prevent attaining “desirable” p values, and rounding off a p value (e.g., stating .054 as .05) (Bruns & Ioannidis, 2016; John, Loewenstein, & Prelec, 2012).”
BackgroundHow puzzling is the forward premium puzzle? A meta-analysis
Zigraiova, D., Havranek, T., Irsova, Z., & Novak, J. (2021) · European Economic Review · doi:10.1016/j.euroecorev.2021.103714 · Cited by 71
Student Employment and Education: A Meta-Analysis
Kroupová, K., Havránek, T., & Iršová, Z. (2024) · Economics of Education Review · doi:10.1016/j.econedurev.2024.102539 · Cited by 72
Increasing the Impact of Materials in and beyond Bio-Nano Science
Björnmalm, M., Faria, M., & Caruso, F. (2016) · Journal of the American Chemical Society · doi:10.1021/jacs.6b08673 · Cited by 68
Reproducibility in the Social Sciences
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022) · Annual Review of Sociology · doi:10.1146/annurev-soc-090221-035954 · Cited by 67
Retraction statements and research malpractice in economics
Cox, A., Craig, R., & Tourish, D. (2018) · Research Policy · doi:10.1016/j.respol.2018.02.016 · Cited by 57
Palpacuer, C., Hammas, K., Duprez, R., Laviolle, B., Ioannidis, J. P. A., & Naudet, F. (2019) · BMC Medicine · doi:10.1186/s12916-019-1409-3 · Cited by 67
Self-citationOpen science, the replication crisis, and environmental public health
Hicks, D. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1962713 · Cited by 65
“The relevance of p-curve analysis to observational studies is controversial: Bruns and Ioannidis (2016) use a simulation study to show that the combination of p-hacking and omitted variable bias can produce a rightskewed p-curve (which would be interpreted as evidence that there was no p-hacking);…”
MethodReporting errors and biases in published empirical findings: Evidence from innovation research
Bruns, S. B., Asanov, I., Bode, R., Dunger, M., Funk, C., Hassan, S. M., Hauschildt, J., Heinisch, D. P., Kempa, K., König, J., et al. (2019) · Research Policy · doi:10.1016/j.respol.2019.05.005 · Cited by 61
Self-citationRelative Risk Aversion: A Meta‐Analysis
Elminejad, A., Havránek, T., & Iršová, Z. (2025) · Journal of Economic Surveys · doi:10.1111/joes.12689 · Cited by 58
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 61
“Questionable research practices bias the p -values (e.g., [15, 18, 19, 32, 33]) and therefore also the truncated density of the original study.”
The limitations of observation studies for decision making regarding drugs efficacy and safety
Gueyffier, F., & Cucherat, M. (2018) · Therapies · doi:10.1016/j.therap.2018.11.001 · Cited by 56
“Moreover, the adjustment-variables set is rarely predefined, leaving the possibility of some p hacking [23, 24].”
Robust empirical calibration of p‐values using observational data
Schuemie, M. J., Hripcsak, G., Ryan, P. B., Madigan, D., & Suchard, M. A. (2016) · Statistics in Medicine · doi:10.1002/sim.6977 · Cited by 53
From Dissent to Democracy: The Promise and Perils of Civil Resistance Transitions
Pinckney, J. C. (2020) · doi:10.1093/oso/9780190097301.001.0001 · Cited by 52
A Tutorial on Hunting Statistical Significance by Chasing N
Szucs, D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01444 · Cited by 45
“Data dredging techniques aim to achieve statistically significant p -value levels and hence, they are also called ‘p-hacking’ (e.g., Bruns and Ioannidis, 2016).”
BackgroundPatel, C. J., Ji, J., Sundquist, J., Ioannidis, J. P. A., & Sundquist, K. (2016) · Scientific Reports · doi:10.1038/srep31308 · Cited by 51
“Furthermore, omitted variable bias, or unaccounted confounding, can result in highly significant but spurious results in observational studies 24 .”
Self-citationvan IJzendoorn, M. H., & Bakermans‐Kranenburg, M. J. (2021) · Attachment & Human Development · doi:10.1080/14616734.2021.1918453 · Cited by 51
“…in small studies unlikely large effect sizes that with or without p-hacking (i.e. datadredging until a “just significant” p-value is produced; Bruns & Ioannidis, 2016) just reach the alpha < 0.05 criterion are taken as established fact instead of being used as bold conjectures for further…”
MethodInfluentialWhen Does Monetary Policy Sway House Prices? A Meta-Analysis
Ehrenbergerová, D., Bajzík, J., & Havránek, T. (2022) · IMF Economic Review · doi:10.1057/s41308-022-00185-5 · Cited by 49
“(2016), Bruns and Ioannidis (2016), Ioannidis et al.”
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01010-1 · Cited by 49
Bruns, S. B., & Stern, D. I. (2018) · Empirical Economics · doi:10.1007/s00181-018-1446-3 · Cited by 48
“Larger values of z indicate smaller p -values and, consequently, higher levels of statistical significance.”
BackgroundInfluentialSelf-citationStatistical data presentation: a primer for rheumatology researchers
Misra, D. P., Zimba, O., & Gasparyan, A. Y. (2020) · Rheumatology International · doi:10.1007/s00296-020-04740-z · Cited by 48
WFSBP guidelines on how to grade treatment evidence for clinical guideline development
Hasan, A., Bandelow, B., Yatham, L. N., Berk, M., Falkai, P., Möller, H.-J., Kasper, S., & WFSBP Guideline Task Force Chairs (2019) · The World Journal of Biological Psychiatry · doi:10.1080/15622975.2018.1557346 · Cited by 44
“External validity is dependent on the internal validity, because if the internal validity of a given trial is poor, the question of its external validity becomes meaningless (Juni et al. 2001; AWMF 2012, 2016; Bruns and Ioannidis 2016).”
BackgroundThe case for preregistering all region of interest (ROI) analyses in neuroimaging research
Gentili, C., Cecchetti, L., Handjaras, G., Lettieri, G., & Cristea, I. A. (2020) · European Journal of Neuroscience · doi:10.1111/ejn.14954 · Cited by 47
“P-hacking could take the form of choosing to report only a subset of ROI analyses out of a suite performed or, upon examining data, adding observations until the effect in an ROI becomes significant (Bruns & Ioannidis, 2016; Poldrack et al., 2017).”
MethodMeasuring Sex Differences and Similarities
Giudice, M. D. (2022) · Focus on sexuality research · doi:10.1007/978-3-030-84273-4_1 · Cited by 47
A Survey of Tasks and Visualizations in Multiverse Analysis Reports
Hall, B. D., Liu, Y., Jansen, Y., Dragicevic, P., Chevalier, F., & Kay, M. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14443 · Cited by 46
Miller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 43
“…to identify the proportions of significant results representing true versus spurious effects (e.g., Simonsohn et al., 2014b), although there is still uncertainty about the exact conditions needed for the valid application of this technique (e.g., Bruns & Ioannidis, 2016; Ulrich & Miller, 2015).”
Identification of and correction for publication biasPreprint
Kasy, M., & Andrews, I. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/49yst · Cited by 42
Erasmus, A., Holman, B., & Ioannidis, J. P. A. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111584 · Cited by 41
Self-citationNew Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
scientist, B. S.-E. D., Mertens, W., & Recker, J. (2020) · Journal of the Association for Information Systems · doi:10.17705/1jais.00629 · Cited by 40
“This can lead to the identification of many correlations that can be mistaken as revealing true relationships (Bruns & Ioannidis, 2016), which can lead to the computation of biased and inconsistent estimations of effects.”
BackgroundA partial solution for the replication crisis in economics
Briggs, W. M. (2023) · Asian Journal of Economics and Banking · doi:10.1108/ajeb-03-2023-0027 · Cited by 39
P‐Hacking , P‐Curves , and the PSM –Performance Relationship: Is There Evidential Value?
Vogel, D., & Homberg, F. (2020) · Public Administration Review · doi:10.1111/puar.13273 · Cited by 37
New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
Mertens, W., & Recker, J. (2023) · Technology, work and globalization · doi:10.1007/978-3-031-38719-7_13 · Cited by 36
p-Hacking and False Discovery in A/B TestingPreprint
Berman, R., Pekelis, L., Scott, A., & den Bulte, C. V. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3204791 · Cited by 36
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0967-6 · Cited by 36
“Second, p -hacking or questionable research practices distort the distribution of p values, and therefore also of conditional probabilities (Bruns & Ioannidis, 2016; Simonsohn et al., 2014a; Ulrich & Miller, 2015; van Aert et al., 2016, 2015), which will bias the effect size estimates of the hybrid…”
Intertemporal substitution in labor supply: A meta-analysis
Elminejad, A., Havránek, T., Horváth, R., & Iršová, Z. (2023) · Review of Economic Dynamics · doi:10.1016/j.red.2023.10.001 · Cited by 35
Identification of and Correction for Publication Bias
Andrews, I., & Kasy, M. (2017) · National Bureau of Economic Research · doi:10.3386/w23298 · Cited by 33
“Bruns and Ioannidis, 2016; Bruns, 2017), for instance, our corrections provide median unbiased estimates for the sum of the parameter of interest and the omitted variables bias. See Section IV.E below. 2Clemens (2017) terms such studies “reproductions,” to distinguish them from “verifications” (cf.”
Bruns, S. B., & Kalthaus, M. (2019) · Research Policy · doi:10.1016/j.respol.2019.103877 · Cited by 31
“…in model selection, which can be used to select models that ‘work’, that is, they provide estimates that fulfill the researcher’s prior and usually confirm the hypothesis of interest, while those models that did not ‘work’ remain unreported (Hendry, 1980; Leamer, 1985; Bruns and Ioannidis, 2016).”
BackgroundMethodSelf-citationMeta‐Regression Models and Observational Research
Bruns, S. B. (2017) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12172 · Cited by 31
Self-citationChurch, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28
“Standard assumptions based on p -values may not hold because of an unethical (but not uncommon) practice known as “ p -hacking” (Bruns and Ioannidis 2016).”
BackgroundReanalyzing Head et al. (2015): investigating the robustness of widespread p -hacking
Hartgerink, C. H. J. (2017) · PeerJ · doi:10.7717/peerj.3068 · Cited by 28
A meta‐analysis on the effects of just‐below versus round prices
Troll, E. S., Frankenbach, J., Friese, M., & Loschelder, D. D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1353 · Cited by 28
Nobel, A., Lizin, S., Brouwer, R., Bruns, S. B., Stern, D. I., & Malina, R. M. (2020) · Environmental Research Letters · doi:10.1088/1748-9326/ab8ec2 · Cited by 28
“While p-hacking describes selective reporting of statistically significant findings at the analysis level within each study (Simmons et al 2011, Bruns and Ioannidis 2016), publication bias describes selective reporting of studies that contain statistically significant findings, while studies with…”
MethodSelf-citationSimonsohn, U., Nelson, L. D., & Simmons, J. P. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213454 · Cited by 27
Spurious precision in meta-analysis of observational research
Iršová, Z., Bom, P. R. D., Havránek, T., & Rachinger, H. J. (2025) · Nature Communications · doi:10.1038/s41467-025-63261-0 · Cited by 26
The logical structure of experiments lays the foundation for a theory of reproducibility
Buzbas, E. O., Devezer, B., & Baumgaertner, B. O. (2023) · Royal Society Open Science · doi:10.1098/rsos.221042 · Cited by 27
“Another example is p-hacking, a statistically invalid form of performing inference to find statistically significant results (Bruns and Ioannidis, 2016; Gelman and Loken, 2013; Munafò et al., 2017).”
BackgroundThe Elasticity of Substitution between Skilled and Unskilled Labor: A Meta-AnalysisPreprint
Havránek, T., Iršová, Z., Laslopová, Ľ., & Zeynalova, O. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/7z2uh · Cited by 26
“The problem is that the mean reported estimates in economics are routinely exaggerated by publication bias (Brodeur et al., 2016; Bruns & Ioannidis, 2016; Christensen & Miguel, 2018; DellaVigna et al., 2019; Blanco-Perez & Brodeur, 2020; Brodeur et al., 2020), often by a factor of 2 or more…”
BackgroundThe impact of touchscreen devices on consumers’ choice confidence and purchase likelihood
Hattula, J. D., Herzog, W., & Dhar, R. (2022) · Marketing Letters · doi:10.1007/s11002-022-09623-w · Cited by 25
Modelling publication bias and p ‐hacking
Moss, J., & Bin, R. D. (2021) · Biometrics · doi:10.1111/biom.13560 · Cited by 24
“(2014a) has been used for phacking as well, but it has been shown to be not reliable (Bruns and Ioannidis, 2016).”
MethodFinancial crime and punishment: A meta‐analysis
de Batz, L., & Kočenda, E. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12580 · Cited by 24
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 23
“…strategy creates bias that is generally known as the file drawer problem (Rosenthal, 1979), and the latter is usually referred to as selective reporting, HARKing (Kerr, 1998) or p-hacking (Bruns & Ioannidis, 2016; Head, Holman, Lanfear, Kahn, & Jennions, 2015; Simonsohn, Nelson, & Simmons, 2014).”
BackgroundCurrent use of effect size or confidence interval analyses in clinical and biomedical research
Amaral, E. D. O. S., & Line, S. R. P. (2021) · Scientometrics · doi:10.1007/s11192-021-04150-3 · Cited by 23
Özdemir, B. C., Bill, R., Okyar, A., Scheiermann, C., Hayoz, S., & Olivier, T. (2025) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2024-010644 · Cited by 22
“33 In other words, by analyzing the data with various assumptions, the chance of finding statistically significant yet spurious results grows.”
Criterion Validity of the Psychopathy Checklist in Legal Contexts: An Updated Meta-Analysis
Holper, L., Ehrbar, C., Fancellu, F., Hauser, N., Habermeyer, E., & Mokros, A. (2025) · Journal of Personality Assessment · doi:10.1080/00223891.2025.2469268 · Cited by 22
“As a right-skewed p-curve has been suggested to potentially indicate p-hacking in observational compared to experimental research (Bruns & Ioannidis, 2016, however, publication bias may still threaten the validity of the updated meta-analyses.”
Khan, S. M., Gull, A., Akhtar, M., Gull, B., Najmi, A. K., Parveen, R., Ali, J., & Khan, S. I. (2025) · Expert Opinion on Drug Delivery · doi:10.1080/17425247.2025.2556979 · Cited by 23
“For instance, testing multiple outcome variables or time points without correcting for multiplicity inflates the Type I error rate (false positives) [158].”
Detecting Evidential Value and p-Hacking With the p-Curve Tool: A Word of Caution
Erdfelder, E., & Heck, D. W. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000383 · Cited by 22
Accounting research and the significance test crisis
Johnstone, D. J. (2021) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2021.102296 · Cited by 22
Ulrich, R., Miller, J. O., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 21
Replicability in Empirical Legal Research
Chin, J. M., & Zeiler, K. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-121620-085055 · Cited by 21
Spurious Precision in Meta-Analysis of Observational ResearchPreprint
Iršová, Z., Bom, P. R. D., Havránek, T., & Rachinger, H. J. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3qp2w · Cited by 21
Greene, J. A., Chinn, C. A., & Deekens, V. M. (2021) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1860992 · Cited by 20
“05 significance level) and to avoid any practices that could be interpreted as p-hacking [Reliable Processes of statistical methods] (Bruns et al., 2016).”
MethodMathur, M. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1667 · Cited by 15
“We simulated forms of p -hacking resembling those in recent simulation studies, 33,43,54,74 and we designed them to be pernicious to SM-step and our proposed method based on theoretical results regarding p -hacking.”
MethodFusaroli, M., Mitchell, J., Rudolph, A., Rocca, E., & Fusaroli, R. (2025) · Drug Safety · doi:10.1007/s40264-025-01628-4 · Cited by 19
“This approach minimises errors when accounting for multiple concurrent biases, strengthens the theoretical foundation of the analysis, and combats p-hacking (arbitrarily altering parameters to produce significant results [62, 63]).”
p-value Problems? An Examination of Evidential Value in Criminology
Wooditch, A., Fisher, R., Wu, X., & Johnson, N. J. (2020) · Journal of Quantitative Criminology · doi:10.1007/s10940-020-09459-5 · Cited by 19
Tempest in a teacup: An analysis of p-Hacking in organizational research
Gupta, A., & Bosco, F. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281938 · Cited by 19
“The limitations are explained in detail by Bruns and Ioannidis [15] and Simonsohn et al.”
BackgroundMethodRaiders of the lost HARK: a reproducible inference framework for big data science
Prosperi, M., Bian, J., Buchan, I. E., Koopman, J. S., Sperrin, M., & Wang, M. (2019) · Humanities and Social Sciences Communications · doi:10.1057/s41599-019-0340-8 · Cited by 18
“…decades (Browman and Skiftesvik, 2011; Mazzola and Deuling, 2013; Head et al., 2015; Lakens, 2015; Bin Abd Razak et al., 2016; Bosco et al., 2016; Bruns and Ioannidis, 2016; Hartgerink et al., 2016; Nissen et al., 2016; Amrhein et al., 2017; Hartgerink, 2017; Hollenbeck and Wright, 2017; Prior…”
BackgroundStrengths and Weaknesses of Meta-Analyses
Corker, K. S. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0006 · Cited by 18
Metadata
- Authors
- Bruns, S. B., & Ioannidis, J. P. A.
- Affiliations
- University of Kassel (DE)
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-17
- Volume
- 11
- Issue
- 2
- Pages
- e0149144
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4757561
- OpenAlex
- W2275632435
- Cited by
- 210
- References
- 44
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 13 · 2023: 15 · 2022: 19 · 2021: 24 · 2020: 37 · 2019: 20 · 2018: 25 · 2017: 23 · 2016: 11
- FWCI
- 24.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Document Server@UHasselt (UHasselt) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, p-hacking, omitted variable bias, confounding, observational studies, null effect, malaria prevalence, economic growth, p-value calibration, significance threshold
- MeSH terms
- Humans, Malaria, Research, Statistics as Topic, Vibration, Prevalence, Sample Size, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.