Stimulus Sampling and Social Psychological Experimentation
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
The authors discuss the problem with failing to sample stimuli in social psychological experimentation. Although commonly construed as an issue for external validity, the authors emphasize how failure to sample stimuli also can threaten construct validity. They note some circumstances where the need for stimulus sampling is less obvious and more obvious, and they discuss some well-known cognitive biases that can contribute to the failure of researchers to see the need for stimulus sampling. Data are presented from undergraduate students (N = 106), graduate students (N = 72), and psychology faculty (N = 48) showing insensitivity to the need for stimulus sampling except when the problem is made rather obvious. Finally, some of the statistical implications of stimulus sampling with particular concern for power, effect size estimates, and data analysis strategies are noted.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In so doing, we sought to address a limitation of prior research, which has sought to draw general conclusions about morality, based on a limited subset of statements which were not created systematically and may be unrepresentative (Wells & Windschitl, 1999).”
From Signaling (in)tolerance: Social evaluation and metaethical relativism and objectivism (2025) · cites this paper as Wells & Windschitl, 1999
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
(1963) · Medical Entomology and Zoology · Cited by 15,085
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195 · cited 1×
“Many of the best thinkers regarding statistical issues in psychological science have discussed stimulus sampling (e.g., Brunswik, 1947; Campbell, 1960; Campbell & Stanley, 1966; Clark, 1973; Cook & Campbell, 1979; Cronbach & Meehl, 1955; Kenny, 1985; Maher, 1978; Rosenthal & Rosnow, 1991).”
BackgroundJudgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
Judgment Under Uncertainty: Heuristics and Biases.
Shafer, G. R., Kahnerman, D., Slovic, P., & Tversky, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288362 · Cited by 5,657
(2004) · Choice Reviews Online · doi:10.5860/choice.41-3182 · Cited by 4,690
Judgment under uncertainty: List of contributors
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Judgment under uncertainty: Causality and attribution
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Essentials of Behavioral Research: Methods and Data Analysis
Rosenthal, R., & Rosnow, R. L. (1984) · Cited by 3,925
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 4×
“However, this use of the representativeness heuristic to make judgments about the lack of a need for stimulus sampling may be just as questionable as making use of the representativeness heuristic for making judgments of probability (as demonstrated by Tversky & Kahneman, 1971 ).”
BackgroundMethodInfluential
Cited by
The C-OAR-SE procedure for scale development in marketing
Rossiter, J. R. (2002) · International Journal of Research in Marketing · doi:10.1016/s0167-8116(02)00097-6 · Cited by 2,477
The Chicago face database: a free stimulus set of faces and norming data
Ma, D. S., Correll, J., & Wittenbrink, B. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0532-5 · Cited by 1,788
Judd, C. M., Westfall, J., & Kenny, D. A. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028347 · Cited by 1,236
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986
Westfall, J., Kenny, D. A., & Judd, C. M. (2014) · Journal of Experimental Psychology: General · doi:10.1037/xge0000014 · Cited by 928
“This neglect of stimulus sampling (Wells & Windschitl, 1999) has led to many inefficiently designed studies.”
BackgroundMeasuring Attraction to Organizations
Highhouse, S., Lievens, F., & Sinar, E. F. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403258403 · Cited by 849
The effect of video game violence on physiological desensitization to real-life violence
Carnagey, N. L., Anderson, C. A., & Bushman, B. J. (2007) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2006.05.003 · Cited by 640
Violent Video Games and Hostile Expectations: A Test of the General Aggression Model
Bushman, B. J., & Anderson, C. A. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/014616702237649 · Cited by 615
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614
“It is possible that some of the observed effects may be specific to the exemplar agent and not other members of that category (Wells & Windschitl, 1999).”
BackgroundWeeks, B. E. (2015) · Journal of Communication · doi:10.1111/jcom.12164 · Cited by 531
Beware of samples! A cognitive-ecological sampling approach to judgment biases.
Fiedler, K. (2000) · Psychological Review · doi:10.1037//0033-295x.107.4.659 · Cited by 482
The role of public exposure in moral and nonmoral shame and guilt.
Smith, R. H., Webster, J. M., Parrott, W. G., & Eyre, H. L. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.83.1.138 · Cited by 473
Rationality for mortals: how people cope with uncertainty
(2009) · Choice Reviews Online · doi:10.5860/choice.46-4133 · Cited by 469
Nisbet, E. C., Cooper, K. E., & Garrett, R. K. (2015) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716214555474 · Cited by 437
Zelenski, J. M., Dopko, R. L., & Capaldi, C. A. (2015) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2015.01.005 · Cited by 428
The Role of Representative Design in an Ecological Approach to Cognition.
Dhami, M. K., Hertwig, R., & Hoffrage, U. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.6.959 · Cited by 408
Violent Video Games: Specific Effects of Violent Content on Aggressive Thoughts and Behavior
Anderson, C. A., Carnagey, N. L., Flanagan, M., Benjamin, A. J., Eubanks, J., & Valentine, J. C. (2004) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(04)36004-1 · Cited by 389
Konijn, E. A., Nije Bijvank, M., & Bushman, B. J. (2007) · Developmental Psychology · doi:10.1037/0012-1649.43.4.1038 · Cited by 372
Theory-Testing, Generalization, and the Problem of External Validity
Lucas, J. W. (2003) · Sociological Theory · doi:10.1111/1467-9558.00187 · Cited by 372
Designing experiments that generalize
Highhouse, S. (2009) · Organizational Research Methods · doi:10.1177/1094428107300396 · Cited by 353
Keep it real: rethinking the primacy of experimental control in cognitive neuroscience
Nastase, S. A., Goldstein, A., & Hasson, U. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117254 · Cited by 344
Bauman, C. W., McGraw, A. P., Bartels, D. M., & Warren, C. (2014) · Social and Personality Psychology Compass · doi:10.1111/spc3.12131 · Cited by 323
Koch, A., Imhoff, R., Dotsch, R., Unkelbach, C., & Alves, H. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000046 · Cited by 309
An own gender bias and the importance of hair in face recognition
Wright, D. B., & Sladden, B. (2003) · Acta Psychologica · doi:10.1016/s0001-6918(03)00052-0 · Cited by 302
Comfortably Numb: Desensitizing Effects of Violent Media on Helping Others
Bushman, B. J., & Anderson, C. A. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02287.x · Cited by 296
Cui, R., Li, J., & Zhang, D. J. (2020) · Management Science · doi:10.1287/mnsc.2018.3273 · Cited by 290
Pachankis, J. E., Clark, K. A., Burton, C. L., Hughto, J. M. W., Bränström, R., & Keene, D. E. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000282 · Cited by 282
What makes a video go viral? An analysis of emotional contagion and Internet memes
Guadagno, R. E., Rempala, D. M., Murphy, S., & Okdie, B. M. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.04.016 · Cited by 279
Bushman, B. J., Bonacci, A. M., van Dijk, M., & Baumeister, R. F. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.84.5.1027 · Cited by 262
Bryanov, K., & Vziatysheva, V. (2021) · PLOS ONE · doi:10.1371/journal.pone.0253717 · Cited by 211
“Generalizability and comparability of the results generated in experiments that use only one message as a stimulus should be enhanced by replications that employ stimulus sampling techniques [63].”
BackgroundCrowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
“Concerns about the potential dependency of findings on the stimuli used to capture them have been raised repeatedly (e.g., Baribault et al., 2018; Campbell & Fiske, 1959; Judd et al., 2012; Monin & Oppenheimer, 2014; Monin et al., 2007; Wells & Windschitl, 1999).”
Fiedler, K. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691616654458 · Cited by 160
“However, although approximating a representative design (Brunswik, 1955; Dhami, Hertwig, & Hoffrage, 2004; Wells & Windschitl, 1999; Westfall, Kenny, & Judd, 2014) serves to foster the external validity or generality of scientific findings, it cannot solve theoretical problems of discriminant…”
BackgroundSchroeder, J., Kardas, M., & Epley, N. (2017) · Psychological Science · doi:10.1177/0956797617713798 · Cited by 129
“As in Experiment 1, we included multiple communicators to increase the generalizability and ecological validity of the experiment (Wells & Windschitl, 1999).”
MethodAvery, D. R., Richeson, J. A., Hebl, M. R., & Ambady, N. (2009) · Journal of Applied Psychology · doi:10.1037/a0016208 · Cited by 129
The (minimal) persuasive advantage of political video over text
Wittenberg, C., Tappin, B. M., Berinsky, A. J., & Rand, D. G. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2114388118 · Cited by 114
“We sought to include a wide variety of policy-relevant clips in order to understand the general effect of video versus text across subject areas (43, 44); to this end, across our two studies, we incorporated a total of 72 persuasive messages spanning a range of topics and varying in the extent to which their content w…”
99% impossible: A valid, or falsifiable, internal meta-analysis.
Vosgerau, J., Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2019) · Journal of Experimental Psychology: General · doi:10.1037/xge0000663 · Cited by 98
Violent video game play impacts facial emotion recognition
Kirsh, S. J., & Mounts, J. R. W. (2007) · Aggressive Behavior · doi:10.1002/ab.20191 · Cited by 95
Dahl, M., & Krog, N. (2018) · European Sociological Review · doi:10.1093/esr/jcy020 · Cited by 79
Song, H., Markowitz, D. M., & Taylor, S. H. (2022) · Journal of Communication · doi:10.1093/joc/jqac017 · Cited by 41
Parent-child joint media engagement in infancy
Padilla‐Walker, L. M., Coyne, S. M., Booth, M. A., Domoff, S. E., Summers, K., Schvaneveldt, E., & Stockdale, L. (2020) · Infancy · doi:10.1111/infa.12355 · Cited by 26
Inducing feelings of ignorance makes people more receptive to expert (economist) opinion
Meyers, E. A., Turpin, M. H., Białek, M., Fugelsang, J. A., & Koehler, D. J. (2020) · Judgment and Decision Making · doi:10.1017/s1930297500008135 · Cited by 23
Charman, S. D., & Carol, R. N. (2012) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2012.07.008 · Cited by 18
Porter, C. L., Stockdale, L. A., Reschke, P., Booth, M., Memmott‐Elison, M. K., & Coyne, S. M. (2022) · Developmental Psychobiology · doi:10.1002/dev.22337 · Cited by 9
Signaling (in)tolerance: Social evaluation and metaethical relativism and objectivism
Moss, D., Montealegre, A., Bush, L. S., Caviola, L., & Pizarro, D. (2025) · Cognition · doi:10.1016/j.cognition.2024.105984 · Cited by 5
Metadata
- Authors
- Wells, G. L., & Windschitl, P. D.
- Venue
- Personality and Social Psychology Bulletin
- Publisher
- SAGE Publications
- Published
- 1999-09
- Volume
- 25
- Issue
- 9
- Pages
- 1115-1125
- ISSN
- 0146-1672 (ISSN-L)
- 1552-7433
- OpenAlex
- W1981458319
- Cited by
- 684
- References
- 21
- FWCI
- 9.00
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.