Encourage Playing with Data and Discourage Questionable Reporting Practices
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Not so long ago, in the early days of experimental psychology, researchers performed their data analyses by hand (or paid people to do these calculations for them).No wonder, they took ample time beforehand to think through which analyses to perform (and especially which not).Exploring data was not really an option.The t value or F value that resulted from the calculations provided a more or less definitive answer.H0 could be rejected or not.During the past decades, we have quickly moved from punch cards, via expensive mainframes, to relatively cheap but powerful personal computers that are fulltime available to individual researchers and students.Nowadays, SPSS and other user-friendly statistical packages make it easy to run all types of analyses, which encourage analyzing your data in every possible way.It is stunningly easy to (re-)run analyses with or without a specific group of participants, with or without an additional between-subject condition, with or without an extra variable in a scale.Indeed, it may be a waste of resources to leave a dataset partly unexplored when SPSS offers all the tools to do so.In many research labs, analyzing results has shifted from spending days to do confirmatory analyses by hand, to spending days engaging in all kinds of data analyses to see "what's in the data."Is this a bad thing?We do not think so, necessarily.Given that you know what you are doing in SPSS, you can learn a lot from exploring a dataset.However, there clearly are risks (see Simmons, Nelson & Simonsohn, 2011), which we will revisit later.This contribution is a commentary on "Playing with Data-Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right" by Sijtsma (2015).In his paper, Klaas Sijtsma makes the important point that prevention of questionable research practices may be more important than detection.He suggests two policy measures.First, make research data and research materials publicly available.Second, encourage researchers to consult methodologists or statisticians for help and a second opinion.In this commentary, we will not reiterate the points made by Klaas Sijtsma.We support his analysis and his policy recommendations.Making research data and research materials publicly available and consulting methodologists and statisticians will help in increasing the transparency of the data analysis phase of the empirical research cycle within psychology.Currently, in too many cases, this phase is the exclusive domain of (small groups of) individual researchers and their personal computers.However, making the data analysis phase transparent to others will not prevent scientific fraud.A person who willingly aims to fool others always will be able to do so, also when sharing (well-faked) data.Nevertheless, it will help in reducing the errors researchers
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Questionable Research Practices Revisited
Fiedler, K., & Schwarz, N. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615612150 · Cited by 279 · cited 1×
“For instance, recently, Fiedler and Schwarz (2014) argued that the high estimates of QRP prevalence reported by John et al. are inflated and therefore misleading.”
Background
Cited by
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Reporting standards for psychological network analyses in cross-sectional data.
Burger, J., Isvoranu, A.-M., Lunansky, G., Haslbeck, J. M. B., Epskamp, S., Hoekstra, R. H. A., Fried, E. I., Borsboom, D., & Blanken, T. F. (2023) · Psychological Methods · doi:10.1037/met0000471 · Cited by 516
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Metacritiques of Upper Echelons Theory: Verdicts and Recommendations for Future Research
Neely, B. H., Lovelace, J. B., Cowen, A. P., & Hiller, N. J. (2020) · Journal of Management · doi:10.1177/0149206320908640 · Cited by 400
Metadata
- Authors
- Wigboldus, D. H. J., & Dotsch, R.
- Venue
- Psychometrika
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-03
- Volume
- 81
- Issue
- 1
- Pages
- 27-32
- ISSN
- 0033-3123 (ISSN-L)
- 1860-0980
- OpenAlex
- W2065437013
- Cited by
- 63
- References
- 5
- FWCI
- 29.57
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Encourage Playing with Data and Discourage Questionable Reporting Practices. https://app.cochat.ai/papers/10.1007/s11336-015-9445-1
- The paper
- Wigboldus, D. H. J., & Dotsch, R. (2016). Encourage Playing with Data and Discourage Questionable Reporting Practices. Psychometrika. https://doi.org/10.1007/s11336-015-9445-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.