Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results
The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, researchers sometimes fail to report those hypotheses that fail to generate positive results (lest reporting such failures hinders publication) [47], even though such practices make the publication of false positives more likely [11].”
From The natural selection of bad science (2016) · cites this paper as [47]
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 2×
“This is consistent with research in the biomedical sciences that shows that published randomized trials continue to exclude negative outcomes or statistically insignificant results despite clinical registry requirements (Chan et al. 2004; Chan and Altman 2005; Rising, Bacchetti, and Bero 2008; Moreno et al. 2009).”
BackgroundResultPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Self-citationWhy We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823 · cited 1×
“For Permissions, please email: journals.permissions@oup.com 306 Miguel 2012; Asendorpf et al. 2013; Franco, Malhotra, and Simonovits 2014; Miguel et al. 2014).1 Despite some obvious benefits, preanalysis plans have been criticized as potentially stifling valuable inductive reasoning and learning…”
BackgroundChan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Registered Reports: A new publishing initiative at Cortex
Chambers, C. D. (2013) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 570
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540 · cited 1×
“For Permissions, please email: journals.permissions@oup.com 306 Miguel 2012; Asendorpf et al. 2013; Franco, Malhotra, and Simonovits 2014; Miguel et al. 2014).1 Despite some obvious benefits, preanalysis plans have been criticized as potentially stifling valuable inductive reasoning and learning…”
BackgroundReshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*
Casey, K., Glennerster, R., & Miguel, E. (2012) · The Quarterly Journal of Economics · doi:10.1093/qje/qje027 · Cited by 453 · cited 1×
“1Recent examples of social science research using preanalysis plans include King et al. (2007), Casey et al. (2012), and Monogan (2013).”
BackgroundType S error rates for classical and Bayesian single and multiple comparison procedures
Gelman, A., & Tuerlinckx, F. (2000) · Computational Statistics · doi:10.1007/s001800000040 · Cited by 380
Research works printed but not linked: 1
- Recent examples of social science research using preanalysis plans include King et al. (2007), Casey et al. (2012), and Monogan (2013). After a new initiative, the journal Cortex became the first psychology journal to review papers solely based on research design and prior to data collection (see Chambers 2013). Since 2013, several other journals in psychology have adopted stricter reporting requirements (e.g., Psychological Science, Experimental Psychology, and Social Psychology).
Cited by
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“Indeed, researchers sometimes fail to report those hypotheses that fail to generate positive results (lest reporting such failures hinders publication) [47], even though such practices make the publication of false positives more likely [11].”
BackgroundComparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Big little lies: a compendium and simulation of p-hacking strategies
Stefan, A., & Schönbrodt, F. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 154
“Franco and colleagues [56] reported a similar prevalence for survey experiments in political science based on empirical data.”
BackgroundTransparency of CHI Research Artifacts: Results of a Self-Reported Survey
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376448 · Cited by 118
“INTRODUCTION Recently, scientific disciplines such as psychology [9], medicine [3], economics [6], and political science [16] have had many published research articles fail to replicate, i.”
BackgroundUnderreporting in Psychology Experiments: Evidence From a Study Registry
Franco, A., Malhotra, N., & Simonovits, G. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 117
Self-citationDeveloping Standards for Post-Hoc Weighting in Population-Based Survey Experiments
Franco, A., Malhotra, N., Simonovits, G., & Zigerell, L. J. (2017) · Journal of Experimental Political Science · doi:10.1017/xps.2017.2 · Cited by 112
“…may cherry-pick estimates based on substantive or statistical significance.5 Thus, in current practice weighting is a researcher degree of freedom akin to the selective reporting of outcome variables, experimental conditions, and model specifications (Franco et al. 2015; Simmons et al. 2011).”
BackgroundSelf-citationWhere Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81
“Fortunately, metascientific efforts to evaluate levels of transparency in specific scientific disciplines have continued (e.g., Bakker et al., 2020; Chin et al., 2021; Christensen et al., 2020; Franco et al., 2015; Fraser et al., 2018; John et al., 2012; Makel et al., 2021; O’Boyle et al., 2017).”
BackgroundPickett, J. T., & Roche, S. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-017-9886-2 · Cited by 82
“While roughly 2% of scientists commit data fraud (Fanelli 2009), a much larger number engage (or have engaged) in selective reporting (Franco et al. 2014, 2015; O’Boyle et al. 2014).”
Questionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 82
“2012), political science (Franco et al. 2015), management (O’Boyle et al.”
BackgroundBakker, B. N., Jaidka, K., Dörr, T., Fasching, N., & Lelkes, Y. (2021) · Journal of Communication · doi:10.1093/joc/jqab031 · Cited by 60
When Efforts to Depolarize the Electorate Fail
Levendusky, M. S. (2018) · Public Opinion Quarterly · doi:10.1093/poq/nfy036 · Cited by 64
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 67
Wuttke, A. (2018) · Politische Vierteljahresschrift · doi:10.1007/s11615-018-0131-7 · Cited by 46
Zigerell, L. J. (2018) · Research & Politics · doi:10.1177/2053168017753862 · Cited by 26
A Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 34
“…Brodeur et al. 2016, 2020; Nelson et al. 2018), low statistical power (Button et al. 2013; Ioannidis et al. 2017), testing hypotheses with low priors (Maniadis et al. 2014; Dreber et al. 2015; Johnson et al 2017), and publication bias (Hedges 1992; Stern & Simes 1997; Franco et al. 2014, 2015).”
Are Voting Treatments Transformative? Expanding Upon and Meta-Analyzing the Evidence
Holbein, J. B., Rangel, M. A., Moore, R., & Croft, M. (2021) · Political Behavior · doi:10.1007/s11109-021-09746-2 · Cited by 24
Ryan, T. J., & Aziz, A. (2020) · The Journal of Politics · doi:10.1086/711133 · Cited by 21
“A recent contribution clearly articulates the perils that arise in using survey weights to convert a sample average treatment effect (SATE) to a population average treatment effect (PATE; Franco et al. 2017).”
BackgroundSurvey Experiments in Sociology
Schachter, A., & Weisshaar, K. (2024) · Annual Review of Sociology · doi:10.1146/annurev-soc-083024-070113 · Cited by 20
Replicability in Empirical Legal Research
Chin, J. M., & Zeiler, K. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-121620-085055 · Cited by 21
Publication Biases in Replication Studies
Berinsky, A. J., Druckman, J. N., & Yamamoto, T. (2020) · Political Analysis · doi:10.1017/pan.2020.34 · Cited by 21
“Regarding the latter point, Franco et al. (2015, 2016) report authorsmake substantial decisions to present limited experimental conditions and outcome variables—these may be frowned upon and thus there is a risk calculus involved in deciding what to present.”
BackgroundPersonality Approaches to Political Behavior
Bakker, B. N. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197541296.013.2 · Cited by 17
Questionable Research Practices and Open Science in Quantitative CriminologyPreprint
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/bwm7s · Cited by 12
Publication Bias in the Social Sciences: A Threat to Scientific Integrity
Malhotra, N. A., & Zigerell, L. J. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0013 · Cited by 11
Self-citationTransparency of CHI Research Artifacts: Results of a Self-Reported SurveyPreprint
Wacharamanotham, C., Eisenring, L., Haroz, S., & Echtler, F. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/3bu6t · Cited by 11
The file drawer problem in social science survey experiments
Moniz, P., Druckman, J. N., & Freese, J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2426937122 · Cited by 13
A Reproduction Analysis of 106 Articles Using Qualitative Comparative Analysis, 2016–2018
Rohlfing, I., Königshofen, L., Krenzer, S., Schwalbach, J., & Bekmuratovna, A. (2020) · PS Political Science & Politics · doi:10.1017/s1049096520001717 · Cited by 8
“Interest in the reproduction of empirical research has been increasing in political science with regard to quantitative methods (Franco, Malhotra, and Simonovits 2015) and, to a lesser degree, case studies and process tracing (Monroe 2018).”
Reducing Political Bias in Political Science Estimates
Zigerell, L. J. (2017) · PS Political Science & Politics · doi:10.1017/s1049096516002389 · Cited by 7
Jensen, N. M. (2017) · Research & Politics · doi:10.1177/2053168017713646 · Cited by 7
“First, scholars can decide to not submit papers with null results to journals, the so called “file drawer problem” (Franco et al., 2015).”
BackgroundInfluentialOutcome Reporting Bias in Randomized Experiments on Substance Use Disorders
Wooditch, A., Sloas, L. B., Wu, X., & Key, A. (2018) · Journal of Quantitative Criminology · doi:10.1007/s10940-018-9388-7 · Cited by 8
An audit of social science survey experiments
Rauf, T., Voelkel, J. G., Druckman, J., & Freese, J. (2025) · Public Opinion Quarterly · doi:10.1093/poq/nfaf052 · Cited by 4
Sponsorship Effects in Online Surveys
Crabtree, C. D., Kern, H., & Pietryka, M. T. (2020) · Political Behavior · doi:10.1007/s11109-020-09620-7 · Cited by 5
“By including complete screenshots, researchers will help avoid the all-too-frequent mistake of failing to report all survey items and experimental treatments (Franco et al. 2015; Gerber et al. 2014).”
BackgroundExamining public perceptions of US campaign finance over time through survey experiments
Goodliffe, J., & Townsend, K. (2024) · Electoral Studies · doi:10.1016/j.electstud.2024.102813 · Cited by 3
Introduction to the Special Issue: Innovations and Current Challenges in Experimental Methods
Jenke, L. (2022) · Political Analysis · doi:10.1017/pan.2021.26 · Cited by 3
“Scholars publishing survey experiments rarely state their weighting procedures and even more rarely provide a justification for weighting or not weighting (Franco et al. 2017), despite this being precisely what the Standards Committee of the Experimental Research Section of the American Political Science Association s…”
InfluentialQuestionable Research Practices and Open Science in Quantitative CriminologyPreprint
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2024) · CrimRxiv · doi:10.21428/cb6ab371.0d3ec44f · Cited by 3
Open with care: transparency and data sharing in civically engaged research
Mitra, A. (2025) · Politics Groups and Identities · doi:10.1080/21565503.2025.2492026 · Cited by 2
Školník, M., Haman, M., & Čopík, J. (2021) · Political Studies Review · doi:10.1177/14789299211047333 · Cited by 1
“The survey experiment is an increasingly used method based on individual data and is used not only in political science but also in other scientific disciplines (Butler, 2020; Dynes, 2020; Franco et al., 2015; Heinzel and Liese, 2021; Zampetakis and Melas, 2021).”
MethodMaćkiewicz, B., Kuś, K., & Krajewski, G. (2026) · Synthese · doi:10.1007/s11229-026-05727-2 · Cited by 1
“Related TESS-based studies show that selective reporting can also operate within published articles, where experimental conditions and outcome variables present in study questionnaires are omitted from the final report (Franco et al., 2015 , 2016 ).”
Preregistration, Data Sharing, and Other Salutary Behaviors
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0011
Experiments in political science
McDermott, R. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-26629-4.00265-3
Priming common European and democratic values does not reduce affective polarization
Canalejo-Molero, Á., Cicchi, L., da Silva, F. F., Garzia, D., Reiljan, A., & Trechsel, A. H. (2026) · European Political Science Review · doi:10.1017/s1755773926100496
The price of convenience : commercial panels and survey data quality in political communication
Jost, P. (2026) · Gutenberg Open Science · doi:10.25358/openscience-15832 · Cited by 1
Steele, B. G., Ross, M. R., Davenport, F. V., & Hall, E. (2026) · doi:10.25675/3.027405
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015
- Volume
- 23
- Issue
- 2
- Pages
- 306-312
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2026769135
- Cited by
- 44
- References
- 20
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 7 · 2020: 6 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 1
- FWCI
- 21.67
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- underreporting, survey experiment, multiple comparisons, preregistration, type I error, statistical reporting, research reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.