Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results
The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, researchers sometimes fail to report those hypotheses that fail to generate positive results (lest reporting such failures hinders publication) [47], even though such practices make the publication of false positives more likely [11].”
From The natural selection of bad science (2016) · cites this paper as [47]
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 2×
“This is consistent with research in the biomedical sciences that shows that published randomized trials continue to exclude negative outcomes or statistically insignificant results despite clinical registry requirements (Chan et al. 2004; Chan and Altman 2005; Rising, Bacchetti, and Bero 2008; Moreno et al. 2009).”
BackgroundResultPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Self-citationWhy We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823 · cited 1×
“For Permissions, please email: journals.permissions@oup.com 306 Miguel 2012; Asendorpf et al. 2013; Franco, Malhotra, and Simonovits 2014; Miguel et al. 2014).1 Despite some obvious benefits, preanalysis plans have been criticized as potentially stifling valuable inductive reasoning and learning…”
BackgroundChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Registered Reports: A new publishing initiative at Cortex
Chambers, C. D. (2013) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 570
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540 · cited 1×
“For Permissions, please email: journals.permissions@oup.com 306 Miguel 2012; Asendorpf et al. 2013; Franco, Malhotra, and Simonovits 2014; Miguel et al. 2014).1 Despite some obvious benefits, preanalysis plans have been criticized as potentially stifling valuable inductive reasoning and learning…”
BackgroundReshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*
Casey, K., Glennerster, R., & Miguel, E. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 453 · cited 1×
“1Recent examples of social science research using preanalysis plans include King et al. (2007), Casey et al. (2012), and Monogan (2013).”
BackgroundType S error rates for classical and Bayesian single and multiple comparison procedures
Gelman, A., & Tuerlinckx, F. (2000) · Computational Statistics · doi:10.1007/s001800000040 · Cited by 380
Research works printed but not linked: 1
- Recent examples of social science research using preanalysis plans include King et al. (2007), Casey et al. (2012), and Monogan (2013). After a new initiative, the journal Cortex became the first psychology journal to review papers solely based on research design and prior to data collection (see Chambers 2013). Since 2013, several other journals in psychology have adopted stricter reporting requirements (e.g., Psychological Science, Experimental Psychology, and Social Psychology).
Cited by
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“Indeed, researchers sometimes fail to report those hypotheses that fail to generate positive results (lest reporting such failures hinders publication) [47], even though such practices make the publication of false positives more likely [11].”
BackgroundComparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Big little lies: a compendium and simulation of p-hacking strategies
Stefan, A., & Schönbrodt, F. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 154
“Franco and colleagues [56] reported a similar prevalence for survey experiments in political science based on empirical data.”
BackgroundTransparency of CHI Research Artifacts: Results of a Self-Reported Survey
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376448 · Cited by 118
“INTRODUCTION Recently, scientific disciplines such as psychology [9], medicine [3], economics [6], and political science [16] have had many published research articles fail to replicate, i.”
BackgroundUnderreporting in Psychology Experiments: Evidence From a Study Registry
Franco, A., Malhotra, N., & Simonovits, G. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 117
Self-citation
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015
- Volume
- 23
- Issue
- 2
- Pages
- 306-312
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2026769135
- Cited by
- 44
- References
- 20
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 7 · 2020: 6 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 1
- FWCI
- 21.67
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- underreporting, survey experiment, multiple comparisons, preregistration, type I error, statistical reporting, research reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.