The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
Abstract
Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition, researchers often engage in analytical practices that capitalize on chance to obtain positive findings, such as p hacking-repeatedly running statistical analyses and only reporting those that are significant-and”
From Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them (2023) · cites this paper as John et al., 2012
In the citing paper
“Even though reporting inconsistencies can be honest mistakes, they have also been categorized as one of several fairly common questionable research practices (QRPs) in psychology (John, Loewenstein, & Prelec, 2012).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as John, Loewenstein, & Prelec, 2012
In the citing paper
“This is in line with the finding of John et al. (2012) that social psychologists admit to more QRPs, find them more applicable to their field, and find them more defensible as compared to other subgroups in psychology (but see also Fiedler & Schwarz, 2015, on this issue).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as John et al. (2012)
In the citing paper
“Such questionable practices are common in psychology (28), and likely pervasive in other disciplines as well.”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [28]
In the citing paper
“A survey of researchers in psychology suggested that practices such as excluding outliers post-hoc, using multiple outcome measures and only reporting results that reached statistical significance, and halting data collection to test for significance and resuming if significance is not found are common (100).”
From Intranasal Oxytocin: Myths and Delusions (2016) · cites this paper as [100]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“…have expressed concerns over failures to replicate published results (Bower & Mayer, 1985; Crabbe, Wahlsten, & Dudek, 1999; Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999; Galak, LeBoeuf, Nelson, & Simmons, 2012; Ioannidis, 2005a, 2005b; Palmer, 2000; Steele, Bass, & Crook, 1999).”
Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias
Warner, S. L. (1965) · Journal of the American Statistical Association · doi:10.1080/01621459.1965.10480775 · Cited by 3,255
Sensitive questions in surveys.
Tourangeau, R., & Yan, T. (2007) · Psychological Bulletin · doi:10.1037/0033-2909.133.5.859 · Cited by 3,057 · cited 1×
“*** p < .0005. large body of research designed to assess the prevalence of sensitive behaviors (Bradburn & Sudman, 1979; de Jong, Pieters, & Fox, 2010; Lensvelt-Mulders, Hox, van der Heijden, & Maas, 2005; Tourangeau & Yan, 2007; Warner, 1965).”
BackgroundTurner, C. F., Ku, L., Rogers, S. M., Lindberg, L. D., Pleck, J. H., & Sonenstein, F. L. (1998) · Science · doi:10.1126/science.280.5365.867 · Cited by 2,046
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 4×
“For example, a meta-analysis of surveys—none of which provided incentives for truthful responding—found that, among scientists from a variety of disciplines, 9.5% of respondents admitted to having engaged in QRPs other than data falsification; the upper-boundary estimate was 33.7% (Fanelli, 2009).”
BackgroundResultInfluentialGenetics of Mouse Behavior: Interactions with Laboratory Environment
Crabbe, J. C., Wahlsten, D., & Dudek, B. C. (1999) · Science · doi:10.1126/science.284.5420.1670 · Cited by 1,666
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Martinson, B. C., Anderson, M. S., & de Vries, R. (2005) · Nature · doi:10.1038/435737a · Cited by 1,096
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670 · cited 1×
“…that many researchers have expressed concerns over failures to replicate published results (Bower & Mayer, 1985; Crabbe, Wahlsten, & Dudek, 1999; Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999; Galak, LeBoeuf, Nelson, & Simmons, 2012; Ioannidis, 2005a, 2005b; Palmer, 2000; Steele,…”
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530
Self-citationFostering integrity in research: Definitions, current knowledge, and future directions
Steneck, N. H. (2006) · Science and Engineering Ethics · doi:10.1007/pl00022268 · Cited by 407
Palmer, A. R. (2000) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.31.1.441 · Cited by 385
Meta-Analysis of Randomized Response Research: Thirty-Five Years of Validation
Lensvelt-Mulders, G. J. L. M., Hox, J. J., van der Heijden, P. G. M., & Maas, C. J. M. (2005) · Sociological Methods & Research · doi:10.1177/0049124104268664 · Cited by 382
How the case against the MMR vaccine was fixed
Deer, B. (2011) · BMJ · doi:10.1136/bmj.c5347 · Cited by 349 · cited 1×
“Although cases of overt scientific misconduct have received significant media attention recently (Altman, 2006; Deer, 2011; Steneck, 2002, 2006), exploitation of the gray area of acceptable practice is certainly much more prevalent, and may be more damaging to the academic enterprise in the long run, than outright fra…”
BackgroundUnpublished results hide the decline effect
Schooler, J. (2011) · Nature · doi:10.1038/470437a · Cited by 333
A case for a duty to feed the hungry: GM plants and the third world
Carter, L. (2007) · Science and Engineering Ethics · doi:10.1007/s11948-006-0006-y · Cited by 265
Ethical Problems in Academic Research
Swazey, J. P., Anderson, M. S., & Ks, L. (1993) · DigitalGeorgetown (Georgetown University Library) · Cited by 243
The Mystery of the Mozart Effect: Failure to Replicate
Steele, K. M., Bass, K. E., & Crook, M. D. (1999) · Psychological Science · doi:10.1111/1467-9280.00169 · Cited by 235
Failure to replicate mood-dependent retrieval
Bower, G. H., & Mayer, J. D. (1985) · Bulletin of the Psychonomic Society · doi:10.3758/bf03329773 · Cited by 183
Schuman, H., Bradburn, N. M., Sudman, S., Blair, E., Locander, W., Miles, C., Singer, E., & Stocking, C. (1980) · Social Forces · doi:10.2307/2577870 · Cited by 178
Correcting the past: Failures to replicate psi.
Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029709 · Cited by 164
de Jong, M. G., Pieters, R. G. M., & Fox, J. (2010) · Journal of Marketing Research · doi:10.1509/jmkr.47.1.14 · Cited by 163
Scaling: A Source Book for Behavioral Scientists
Fox, J. D., & Maranell, G. M. (1975) · The Canadian Journal of Sociology · doi:10.2307/3340023 · Cited by 154
Academic economists behaving badly? A survey on three areas of unethical behavior
List, J. F., Bailey, C., Euzent, P. J., & Martin, T. (2001) · Economic Inquiry · doi:10.1111/j.1465-7295.2001.tb00058.x · Cited by 143
Sovacool, B. K. (2008) · Journal of Bioethical Inquiry · doi:10.1007/s11673-008-9113-6 · Cited by 116
Psychology must learn a lesson from fraud case
Wicherts, J. M. (2011) · Nature · doi:10.1038/480007a · Cited by 86
Academic economists behaving badly? A survey on three areas of unethical behavior
List, J. A. (2001) · Economic Inquiry · doi:10.1093/ei/39.1.162 · Cited by 75
Authors' reports about research integrity problems in clinical trials
Gardner, W., Lidz, C. W., & Hartwig, K. C. (2005) · Contemporary Clinical Trials · doi:10.1016/j.cct.2004.11.013 · Cited by 69
A survey of newly appointed consultants' attitudes towards research fraud
Geggie, D. (2001) · Journal of Medical Ethics · doi:10.1136/jme.27.5.344 · Cited by 62
How Prevalent Is Fraud? That's a Million-Dollar Question
Marshall, E. (2000) · Science · doi:10.1126/science.290.5497.1662 · Cited by 61
Henry, D. A., Hill, S. R., Doran, E., Newby, D., Henderson, K. M., Maguire, J. M., Stokes, B., Kerridge, I., McNeill, P. M., Day, R. O., et al. (2005) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2005.tb06813.x · Cited by 53
The road to fraud starts with a single step
Crocker, J. (2011) · Nature · doi:10.1038/479151a · Cited by 41
Sterba, S. K. (2006) · Ethics & Behavior · doi:10.1207/s15327019eb1604_3 · Cited by 20
Correcting the Past: Failures to Replicate PsiPreprint
Galak, J., LeBoeuf, R. A., Nelson, L. D., et al. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2001721 · Cited by 13
Fickle Mice Highlight Test Problems
Enserink, M. (1999) · Science · doi:10.1126/science.284.5420.1599a · Cited by 12
Two stories about evolution on The New York Times and a strange “editorial balance”
Eldredge, N. (2006) · Journal of Science Communication · doi:10.22323/2.05020302 · Cited by 8
The Economics of Scientific Misconduct
Lacetera, N., & Zirulia, L. (2011) · Journal of Law, Economics, and Organization · doi:10.1093/jleo/ewp031 · Cited by 5
Research works printed but not linked: 3
- Guttman L. L., 1974, Scaling: A sourcebook for behavioral scientists, 142
- Steneck N. H., 2002, Investigating Research Integrity: Proceedings of the First ORI Research Conference on Research Integrity, 1
- Swazey J. P., 1993, American Scientist, 81, 542
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
“In a hypothesis-testing study powered to detect an effect on the primary outcome measure, data on secondary outcomes are used to evaluate additional effects of the intervention, but subsequent statistical analysis of secondary outcome measures may be underpowered, making results and interpretation less reliable [88,93…”
Background
Metadata
- Authors
- John, L. K., Loewenstein, G., & Prelec, D.
- Affiliations
- Harvard University (US)
- Carnegie Mellon University (US)
- Sloan School of Management and Departments of Economics and Brain & Cognitive Sciences, Massachusetts Institute of Technology
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2012-04-16
- Volume
- 23
- Issue
- 5
- Pages
- 524-532
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2100267603
- Cited by
- 2,239
- References
- 39
- Citations by year
- 2026: 85 · 2025: 120 · 2024: 140 · 2023: 219 · 2022: 234 · 2021: 284 · 2020: 220 · 2019: 239 · 2018: 177 · 2017: 147 · 2016: 128 · 2015: 90 · 2014: 77 · 2013: 42 · 2012: 33
- FWCI
- 201.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Association for Psychological Science
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- questionable research practices, prevalence, truth-telling incentives, research misconduct, research norms, psychologists
- MeSH terms
- Data Collection, Humans, Motivation, Psychology, Research, Truth Disclosure, Scientific Misconduct
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.