CoChat

Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results

Franco, Annie, Malhotra, Neil, Simonovits, Gabor · 2015 · Political Analysis

The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.

Abstract

The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.

Overview

Integrity signals

Registry

Crossref · 10.1093/pan/mpv006

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 19 references

Research works printed but not linked: 1

  1. Recent examples of social science research using preanalysis plans include King et al. (2007), Casey et al. (2012), and Monogan (2013). After a new initiative, the journal Cortex became the first psychology journal to review papers solely based on research design and prior to data collection (see Chambers 2013). Since 2013, several other journals in psychology have adopted stricter reporting requirements (e.g., Psychological Science, Experimental Psychology, and Social Psychology).

Cited by

See all 44

Metadata

Authors
Franco, A., Malhotra, N., & Simonovits, G.
Affiliations
Stanford University (US)
Corresponding author
Malhotra, Neil
Venue
Political Analysis
Published
2015
Volume
23
Issue
2
Pages
306-312
ISSN
  • 1047-1987 (ISSN-L)
  • 1476-4989
OpenAlex
W2026769135
Cited by
44
References
20
Citations by year
2026: 3 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 7 · 2020: 6 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 1
FWCI
21.67
Type
journal article
Language
en
Open access
closed
Locations
  • Political Analysis (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Open Research (Auckland University of Technology) (submittedVersion)
Topics
  • Social Sciences › Political Science and International Relations › Political Science Research and Education
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords
underreporting, survey experiment, multiple comparisons, preregistration, type I error, statistical reporting, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.