Research Practices That Can Prevent an Inflation of False-Positive Rates
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Recent studies have indicated that research practices in psychology may be susceptible to factors that increase false-positive rates, raising concerns about the possible prevalence of false-positive findings. The present article discusses several practices that may run counter to the inflation of false-positive rates. Taking these practices into account would lead to a more balanced view on the false-positive issue. Specifically, we argue that an inflation of false-positive rates would diminish, sometimes to a substantial degree, when researchers (a) have explicit a priori theoretical hypotheses, (b) include multiple replication studies in a single paper, and (c) collect additional data based on observed results. We report findings from simulation studies and statistical evidence that support these arguments. Being aware of these preventive factors allows researchers not to overestimate the pervasiveness of false-positives in psychology and to gauge the susceptibility of a paper to possible false-positives in practical and fair ways.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Stereotype threat and the intellectual test performance of African Americans.
Steele, C. M., & Aronson, J. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.69.5.797 · Cited by 9,020 · cited 2×
“For example, in research on stereotype threat, it is theorized that people experience anxiety or concern in a situation where they have the potential to confirm a negative stereotype about their social group (Steele & Aronson, 1995).”
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“…rates, which make it difficult to draw valid and reliable scientific conclusions (Bakker & Wicherts, 2011; Fiedler, 2011; Garcia-Perez, 2012; Ioannidis, 2005; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011; Vul, Harris, Winkielman, & Pashler, 2009; Young, Ioannidis, &…”
Stereotype Threat and Women's Math Performance
Spencer, S. J., Steele, C. M., & Quinn, D. M. (1999) · Journal of Experimental Social Psychology · doi:10.1006/jesp.1998.1373 · Cited by 3,435 · cited 1×
“Based on this theory, researchers can make the prediction that when a stereotype about expected performance is made salient, individuals who belong to negatively-stereotyped groups will perform more poorly than they otherwise would (Spencer, Steele, & Quinn, 1999; Steele & Aronson, 1995).”
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907 · cited 1×
“The crucial point is that strong theories must be explicit about findings that cannot occur (Platt, 1964).”
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · cited 1×
“In fact, we agree with the previous literature that it is important to be aware of the susceptibility of current research practices to the inflation of false-positive rates (Bakker, van Dijk, & Wicherts, 2012; Fiedler, 2011; Simmons et al., 2011; Vul et al., 2009).”
Type I and Type II error concerns in fMRI research: re-balancing the scale
Lieberman, M. D., & Cunningham, W. A. (2009) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsp052 · Cited by 1,342 · cited 1×
“Researchers have argued that such failures may reduce the chances for scientific discovery, and that neglect of false-negatives can limit novel and generative research endeavours (see Fiedler et al., 2012; Lieberman & Cunningham, 2009).”
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 2×
“We fully agree that researcher degrees of freedom (and other incentive factors, as discussed by Nosek et al., 2012) can inflate Type-1 error rates and that some practices may have led to publications that actually report false-positive findings.”
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 1×
“It has been pointed out, however, that researchers sometimes present post-hoc hypotheses in a research report as if they were, in fact, a priori hypotheses (John et al., 2012; Okada & Shimokido, 2001; see also Fanelli, 2012).”
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 2×
“In a multi-study paper, researchers commonly conceptually replicate their primary findings in subsequent studies (Schmidt, 2009).”
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 1×
“…because it is inconsistent with cumulative prior evidence), the susceptibility of the respective research to false positives should be assessed through other means such as replication studies (see also the concept of confirmatory studies; Wagenmakers et al., 2012), as discussed in the next section.”
In All Likelihood: Statistical Modelling and Inference Using Likelihood
Pawitan, Y. (2001) · doi:10.1093/oso/9780198507659.001.0001 · Cited by 874
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 1×
“In fact, we agree with the previous literature that it is important to be aware of the susceptibility of current research practices to the inflation of false-positive rates (Bakker, van Dijk, & Wicherts, 2012; Fiedler, 2011; Simmons et al., 2011; Vul et al., 2009).”
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · cited 1×
“…power (i.e., the probability of rejecting the null hypothesis when the null hypothesis is false) for each study in that paper, and multiplied the power probabilities to estimate the probability that the observed findings would all reject the null hypothesis in a row (Ioannidis & Trikalinos, 2007).”
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012) · doi:10.2139/ssrn.2160588 · Cited by 680
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 1×
“…susceptible to a variety of factors that increase false-positive rates, which make it difficult to draw valid and reliable scientific conclusions (Bakker & Wicherts, 2011; Fiedler, 2011; Garcia-Perez, 2012; Ioannidis, 2005; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011;…”
Legault, L., & Inzlicht, M. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0030426 · Cited by 210
Bite-size science and its undesired side effects
Bertamini, M., & Munafò, M. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611429353 · Cited by 62 · cited 1×
“…opinion on the pros and cons of single-study papers, but in light of the current paper, one point that has not been made in the ongoing discussion (Bertamini & Munafo, 2012; Ledgerwood & Sherman, 2012) is that researchers should pay more attention to other factors than the number of studies that…”
Cited by
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
Pre-registration in social psychology—A discussion and suggested template
van 't Veer, A. E., & Giner-Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 439
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 393
Metadata
- Authors
- Murayama, K., Pekrun, R., & Fiedler, K.
- Venue
- Personality and Social Psychology Review
- Publisher
- SAGE Publications
- Published
- 2013-08-21
- Volume
- 18
- Issue
- 2
- Pages
- 107-118
- ISSN
- 1088-8683 (ISSN-L)
- 1532-7957
- OpenAlex
- W2155478468
- Cited by
- 129
- References
- 26
- FWCI
- 18.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.