The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
Open accessAbstract
Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Designing Social Inquiry: Scientific Inference in Qualitative Research
King, G., Keohane, R. O., & Verba, S. (1994) · Princeton University Press · doi:10.1515/9781400821211 · Cited by 7,225
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“This kind of concern has led some to worry that “most current published research findings are false” (Ioannidis 2005).”
BackgroundExperimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Handbook of development economics
de Melo, J. (1992) · Journal of Development Economics · doi:10.1016/0304-3878(92)90010-7 · Cited by 1,645
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Cited by
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments
Guess, A., & Coppock, A. (2020) · British Journal of Political Science · doi:10.1017/s0007123418000327 · Cited by 372
The Populist Style in American Politics: Presidential Campaign Discourse, 1952–1996
Bonikowski, B., & Gidron, N. (2016) · Social Forces · doi:10.1093/sf/sov120 · Cited by 342
Machine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Metadata
- Authors
- Humphreys, M., Sanchez de la Sierra, R., & van der Windt, P.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 1
- Pages
- 1-20
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095635841
- Cited by
- 298
- References
- 41
- FWCI
- 21.79
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration. https://app.cochat.ai/papers/10.1093/pan/mps021
- The paper
- Humphreys, M., Sanchez de la Sierra, R., & van der Windt, P. (2013). Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration. Political Analysis. https://doi.org/10.1093/pan/mps021
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.