The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Generalizability of heterogeneous treatment effect estimates across samples
Open accessAbstract
Significance: In experiments, the degree to which results generalize to other populations depends critically on the degree of treatment effect heterogeneity. We replicated 27 survey experiments (encompassing 101,745 individual survey responses) originally conducted on nationally representative samples using online convenience samples, finding very high correspondence despite obvious differences in sample composition. We contend this pattern is due to low treatment effect heterogeneity in these types of standard social science survey experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841 · cited 1×
“The response to this evidence should not be that any convenience sample can be used to study any treatment without concern about generalizability (23) but rather that debates about generalizability and replication must focus on the underlying causes of replication and nonreplication, among these most importantly the v…”
BackgroundThe Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Self-citationStriking a Responsive Chord: How Political Ads Motivate and Persuade Voters by Appealing to Emotions
Brader, T. (2005) · American Journal of Political Science · doi:10.1111/j.0092-5853.2005.00130.x · Cited by 972
Statistical Adjustment of Data
Anderson, R. L., & Deming, W. E. (1944) · Southern Economic Journal · doi:10.2307/1052578 · Cited by 769
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 686
Cambridge Handbook of Experimental Political Science
(2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452 · Cited by 611
Weinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
Social Science Methodology: A Unified Framework
Gerring, J. (2012) · Cited by 515
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · cited 1×
“A complete description of each experiment and replication procedures are available in the original papers and their supplementary materials (12, 13).”
MethodSelf-citationGeneralizing Evidence From Randomized Clinical Trials to Target Populations: The ACTG 320 Trial
Cole, S. R., & Stuart, E. A. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq084 · Cited by 486
Craig, M. A., & Richeson, J. A. (2014) · Personality and Social Psychology Bulletin · doi:10.1177/0146167214524993 · Cited by 471
Nicholson, S. P. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00541.x · Cited by 455
Does Media Coverage of Partisan Polarization Affect Political Attitudes?
Levendusky, M., & Malhotra, N. (2016) · Political Communication · doi:10.1080/10584609.2015.1038455 · Cited by 419
Dynamic Public Opinion: Communication Effects over Time
CHONG, D., & DRUCKMAN, J. N. (2010) · American Political Science Review · doi:10.1017/s0003055410000493 · Cited by 395
Transue, J. E. (2007) · American Journal of Political Science · doi:10.1111/j.1540-5907.2007.00238.x · Cited by 383
The Growth and Development of Experimental Research in Political Science
DRUCKMAN, J. N., GREEN, D. P., KUKLINSKI, J. H., & LUPIA, A. (2006) · American Political Science Review · doi:10.1017/s0003055406062514 · Cited by 374
Evaluation of regression procedures for methods comparison studies
Linnet, K. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.3.424 · Cited by 323
Caprariello, P. A., & Reis, H. T. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0030953 · Cited by 295
McGinty, E. E., Webster, D. W., & Barry, C. L. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2013.13010014 · Cited by 289
Hiscox, M. J. (2006) · International Organization · doi:10.1017/s0020818306060255 · Cited by 259
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251
Persuasion and Resistance: Race and the Death Penalty in America
Peffley, M., & Hurwitz, J. (2007) · American Journal of Political Science · doi:10.1111/j.1540-5907.2007.00293.x · Cited by 230
Hartman, E., Grieve, R., Ramsahai, R. R., & Sekhon, J. S. (2015) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12094 · Cited by 203
Pedulla, D. S. (2014) · Social Psychology Quarterly · doi:10.1177/0190272513506229 · Cited by 200
Does Regression Produce Representative Estimates of Causal Effects?
Aronow, P. M., & Samii, C. (2016) · American Journal of Political Science · doi:10.1111/ajps.12185 · Cited by 192
Metatheory in Social Science: Pluralisms and Subjectivities.
Holy, L., Fiske, D. W., & Shweder, R. A. (1988) · Man · doi:10.2307/2803087 · Cited by 192
Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations
Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2015.1060282 · Cited by 162
Believers' estimates of God's beliefs are more egocentric than estimates of other people's beliefs
Epley, N., Converse, B. A., Delbosc, A., Monteleone, G., & Cacioppo, J. T. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0908374106 · Cited by 113
Nguyen, T. Q., Ebnesajjad, C. F., Cole, S. R., & Stuart, E. A. (2017) · The Annals of Applied Statistics · doi:10.1214/16-aoas1001 · Cited by 105
Statistical adjustment of data
Oppermann, R. H. (1944) · Journal of the Franklin Institute · doi:10.1016/s0016-0032(44)90975-0 · Cited by 101
Economists and Public Opinion: Expert Consensus and Economic Policy Judgments
Johnston, C. D., & Ballard, A. M. O. (2014) · The Journal of Politics · doi:10.1086/684629 · Cited by 93
Piazza, J. A. (2014) · Political Psychology · doi:10.1111/pops.12190 · Cited by 91
Samii, C., & Aronow, P. M. (2011) · Statistics & Probability Letters · doi:10.1016/j.spl.2011.10.024 · Cited by 74
Jacobsen, R., Snyder, J. W., & Saultz, A. (2014) · American Journal of Education · doi:10.1086/678136 · Cited by 70
Coppock, A., & GREEN, D. P. (2014) · Political Science Research and Methods · doi:10.1017/psrm.2014.10 · Cited by 54
Self-citationNew Directions for Experimental Work in International Relations
McDermott, R. (2011) · International Studies Quarterly · doi:10.1111/j.1468-2478.2011.00656.x · Cited by 49
Assessing the Breadth of Framing Effects
Hopkins, D. J., & Mummolo, J. (2017) · Quarterly Journal of Political Science · doi:10.1561/100.00015139 · Cited by 38
Murtagh, L., Gallagher, T. H., Andrew, P., & Mello, M. M. (2012) · Health Affairs · doi:10.1377/hlthaff.2012.0185 · Cited by 37
Onset and Offset Deservingness: The Case of Home Foreclosures
Brandt, M. J. (2012) · Political Psychology · doi:10.1111/j.1467-9221.2012.00931.x · Cited by 24
Hillary Rodham Versus Hillary Clinton: Consequences of Surname Choice in Marriage
Shafer, E. F. (2017) · Gender Issues · doi:10.1007/s12147-016-9182-5 · Cited by 18
Undermining Optimistic Denial Reactions to Domestic and Campus Emergency Warning Messages
Thompson, S. C. G., & Schlehofer, M. M. (2014) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12024 · Cited by 9
Cited by
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
Self-citationOnline panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Social Media, Web, and Panel Surveys: Using Non‐Probability Samples in Social and Policy Research
Lehdonvirta, V., Oksanen, A., Räsänen, P., & Blank, G. (2021) · Policy & Internet · doi:10.1002/poi3.238 · Cited by 393
Addressing climate change with behavioral science: A global intervention tournament in 63 countries
Vlasceanu, M., Doell, K. C., Bak-Coleman, J. B., Todorova, B., Berkebile-Weinberg, M. M., Grayson, S. J., Patel, Y., Goldwert, D., Pei, Y., Chakroff, A., et al. (2024) · Science Advances · doi:10.1126/sciadv.adj5778 · Cited by 269
CLIFFORD, S., SHEAGLEY, G., & PISTON, S. (2021) · American Political Science Review · doi:10.1017/s0003055421000241 · Cited by 254
Metadata
- Authors
- Coppock, A., Leeper, T. J., & Mullinix, K. J.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2018-11-16
- Volume
- 115
- Issue
- 49
- Pages
- 12441-12446
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W2901206377
- Cited by
- 445
- References
- 44
- FWCI
- 21.57
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.