The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
Open accessAbstract
Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report” of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but nonbinding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but nonbinding registration are discussed, the principal advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Designing Social Inquiry: Scientific Inference in Qualitative Research
King, G., Keohane, R. O., & Verba, S. (1994) · Princeton University Press · doi:10.1515/9781400821211 · Cited by 7,225
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“This kind of concern has led some to worry that “most current published research findings are false” (Ioannidis 2005).”
BackgroundExperimental Analysis of Neighborhood Effects
Kling, J. R., Liebman, J. B., & Katz, L. F. (2007) · Econometrica · doi:10.1111/j.1468-0262.2007.00733.x · Cited by 2,719
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Handbook of development economics
de Melo, J. (1992) · Journal of Development Economics · doi:10.1016/0304-3878(92)90010-7 · Cited by 1,645
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
A Test for Superior Predictive Ability
Hansen, P. R. (2005) · Journal of Business & Economic Statistics · doi:10.1198/073500105000000063 · Cited by 1,230
Using Randomization in Development Economics Research: A Toolkit
Duflo, E., Glennerster, R., & Kremer, M. (2006) · National Bureau of Economic Research · doi:10.3386/t0333 · Cited by 1,191
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“There are other benefits to registration, including the ability to assess the population of studies that are never completed or never published. Here, however, we focus on the benefits of ex ante specification of analysis plans. (5)See, for example, Lancet (2010). A mechanism to report observational studies already ex…”
BackgroundRethinking Social Inquiry: Diverse Tools, Shared StandardsPreprint
Brady, H. E., & Collier, D. (2010) · SSRN Electronic Journal · Cited by 871
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
Rethinking Social Inquiry: Diverse Tools, Shared Standards
(2010) · doi:10.5040/9798216385530 · Cited by 405
Type S error rates for classical and Bayesian single and multiple comparison procedures
Gelman, A., & Tuerlinckx, F. (2000) · Computational Statistics · doi:10.1007/s001800000040 · Cited by 380 · cited 6×
“2 But as highlighted above the problem extends beyond this type of testing and could obtain for example in approaches that focus on “Type S” errors, seeking to make statements of the form “θ1 > θ2, with confidence” (see for example Gelman and Tuerlinckx (2000)). This kind of concern has led some to worry that “most cu…”
BackgroundMethodInfluentialKing, G., Gakidou, E., Imai, K., Lakin, J., Moore, R. T., Nall, C., Ravishankar, N., Vargas, M., Téllez-Rojo, M. M., Ávila, J. E. H., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60239-7 · Cited by 324
Fearon, J. D., Humphreys, M., & Weinstein, J. M. (2009) · American Economic Review · doi:10.1257/aer.99.2.287 · Cited by 314
Self-citationGerber, A., & Malhotra, N. (2008) · Quarterly Journal of Political Science · doi:10.1561/100.00008024 · Cited by 284
Testing for Publication Bias in Political Science
Gerber, A. S., GREEN, D. P., & Nickerson, D. W. (2001) · Political Analysis · doi:10.1093/oxfordjournals.pan.a004877 · Cited by 162
The Oregon Health Insurance Experiment: Evidence from the First Year +
Finkelstein, A. N., Taubman, S. L., Wright, B. J., Mira Bernstein, A., B, J. G. A., Newhouse, J. P., ALLEN, H. L., & D, K. B. A. (2012) · Cited by 109
A Test for Superior Predictive Ability
Hansen, P. R. (2004) · doi:10.2139/ssrn.264569 · Cited by 99
Report finds massive fraud at Dutch universities
Callaway, E. (2011) · Nature · doi:10.1038/479015a · Cited by 88
Researcher Incentives and Empirical MethodsPreprint
Glaeser, E. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.934557 · Cited by 73
Should protocols for observational research be registered?
Lancet, T. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60148-1 · Cited by 72
Revéiz, L., Chan, A., Krleža-Jerić, K., Granados, C., Pinart, M., Etxeandia‐Ikobaltzeta, I., Rada, D., Martínez‐Alonso, M., Bonfill, X., & Cardona, A. F. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012484 · Cited by 46
Empowering Women: Evidence from a Field Experiment in Afghanistan
Beath, A., Christia, F., & Сергеевич, Е. Р. (2012) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-6269 · Cited by 42
Walking the talk: the need for a trial registry for development interventions
Rasmussen, O. D., Malchow‐Møller, N., & Andersen, T. B. (2011) · Journal of Development Effectiveness · doi:10.1080/19439342.2011.605160 · Cited by 42
Should protocols for observational research be registered?
Rushton, L. (2010) · Occupational and Environmental Medicine · doi:10.1136/oem.2010.056846 · Cited by 36
Researcher Incentives and Empirical Methods
Glaeser, E. L. (2006) · National Bureau of Economic Research · doi:10.3386/t0329 · Cited by 34
The Quest For Standards: King, Keohane, And Verba'S Designing Social Inquiry
Collier, D., Seawright, J., & Munck, G. L. (2010) · Rowman & Littlefield Publishers eBooks · doi:10.5040/9798216385530.ch-002 · Cited by 34
Angelis, C. D. D. (2005) · Canadian Medical Association Journal · doi:10.1503/cmaj.050600 · Cited by 32
The Perils of Randomization Checks in the Analysis of Experiments
Mutz, D. C., & Pemantle, R. (2012) · ScholarlyCommons (University of Pennsylvania) · Cited by 31
RESHAPING INSTITUTIONS: EVIDENCE ON EXTERNAL AID AND LOCAL COLLECTIVE ACTION
Casey, K., Glennerster, R., Miguel, E. A., & Poverty, A. L. J. (2011) · Cited by 28
Clinical trial registration: looking back and moving ahead
Laine, C. M., Horton, R., DeAngelis, C. D., Drazen, J. M., Frizelle, F., Godlee, F., Haug, C. J., Hébert, P. C., Kotzin, S., Marušić, A., et al. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.070753 · Cited by 20
Impact evaluation of infrastructure interventions
Hansen, H., Andersen, O. W., & White, H. (2011) · Journal of Development Effectiveness · doi:10.1080/19439342.2011.547659 · Cited by 17
Researcher Incentives and Empirical Methods
GLAESER, E. L. (2008) · The Foundations of Positive and Normative Economics · doi:10.1093/acprof:oso/9780195328318.003.0013 · Cited by 15
Reshaping Institutions: Evidence on External Aid and Local Collective Action (IGC Working Paper)
Casey, K., Glennerster, R., Jameel, A., & Miguel, E. A. (2011) · Cited by 6
Cited by
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments
Guess, A., & Coppock, A. (2020) · British Journal of Political Science · doi:10.1017/s0007123418000327 · Cited by 372
The Populist Style in American Politics: Presidential Campaign Discourse, 1952–1996
Bonikowski, B., & Gidron, N. (2016) · Social Forces · doi:10.1093/sf/sov120 · Cited by 342
Machine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Metadata
- Authors
- Humphreys, M., Sanchez de la Sierra, R., & van der Windt, P.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 1
- Pages
- 1-20
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095635841
- Cited by
- 298
- References
- 41
- FWCI
- 21.79
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.