The DOI matches its Crossref record; 68 of the 72 references it cites were found in a registry or an index.
Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires.
Abstract
The rapid growth of the Internet provides a wealth of new research opportunities for psychologists. Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N = 361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods. It is concluded that Internet methods can contribute to many areas of psychology.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The proportion of females (62%) to males in this study was more representative than in traditional survey methods, where women often comprise 71% to 77% of the participants [39]”
From Effects of Mock Facebook Workday Comments on Public Perception of Professional Credibility: A Field Study in Canada (2019) · cites this paper as [39]
In the citing paper
“Self-presentation is an unlikely confound in Study 1 because the motivation to self-present is absent when participants feel anonymous (Paulhus, 1984(Paulhus, , 1991) ) as they do in anonymous online studies such as the present one, where they don't provide identifying information (Gosling et al., 2004).”
From Cultural norm fulfillment, interpersonal belonging, or getting ahead? A large-scale cross-cultural test of three perspectives on the function of self-esteem. (2015) · cites this paper as Gosling et al., 2004
In the citing paper
“As described in the Appendix, the reliability and validity of internet data collection has become increasingly clear in recent years.”
From Spacing Effects in Learning: A Temporal Ridgeline of Optimal Retention (2008) · cites this paper as Gosling et al., 2004
In the citing paper
“The validity of internet testing has been well supported in recent reviews (Gosling et al., 2004), with excellent correspondence between results obtained with internet samples are results obtained in the laboratory (e.g., Birnbaum, 1999;Krantz & Dalal, 2000;McGraw, Tew, & Williams, 2000;Reips, 2002).”
From Spacing Effects in Learning: A Temporal Ridgeline of Optimal Retention (2008) · cites this paper as Gosling et al., 2004
In the citing paper
“Furthermore, previous research has suggested populations for online studies and Facebook are quite representative [23], [24].”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [23]
In the citing paper
“Still, the sample sizes and availability of demographic information afforded by social media bring us closer to a more ideal representative sample [99].”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [99]
Citations
Cites
Mail and internet surveys : the tailored design method
Dillman, D. A. (2007) · Cited by 9,587
(1999) · Cited by 7,995
Self-citationInternet paradox: A social technology that reduces social involvement and psychological well-being?
Kraut, R., Patterson, M., Lundmark, V., Kiesler, S., Mukophadhyay, T., & Scherlis, W. (1998) · American Psychologist · doi:10.1037/0003-066x.53.9.1017 · Cited by 4,793
Handbook of Personality : Theory and Research
Pervin, L. A. (1992) · Cited by 4,543
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041
Mail and Internet Surveys: The Tailored Design Method
Chisnall, P. M. (2007) · Journal of Advertising Research · doi:10.2501/s0021849907070237 · Cited by 3,463
Gender differences in personality traits across cultures: Robust and surprising findings.
Costa, P. T., Terracciano, A., & McCrae, R. R. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.81.2.322 · Cited by 2,986
Kraut, R. E., Kiesler, S. B., Boneva, B. S., Cummings, J. N., Helgeson, V. S., & Crawford, A. M. (2002) · Journal of Social Issues · doi:10.1111/1540-4560.00248 · Cited by 2,367
Gender differences in personality: A meta-analysis.
Feingold, A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.3.429 · Cited by 2,240
Handbook of Personality Theory and Research.
Schwartz, M. L., Borgatta, E. F., & Lambert, W. W. (1970) · American Sociological Review · doi:10.2307/2093310 · Cited by 2,182
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.3.515 · Cited by 2,111
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,086
Turner, C. F., Ku, L. C., Rogers, S., Lindberg, L., Pleck, J. H., & Sonenstein, F. L. (1998) · Science · doi:10.1126/science.280.5365.867 · Cited by 2,048
Benet-Martínez, V., & John, O. P. (1998) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.75.3.729 · Cited by 1,996
Self-citationThe do re mi's of everyday life: The structure and personality correlates of music preferences.
Rentfrow, P. J., & Gosling, S. D. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.84.6.1236 · Cited by 1,695
Self-citationEffects of Fear and Anger on Perceived Risks of Terrorism
Lerner, J. S., Gonzalez, R. M., Small, D. A., & Fischhoff, B. (2003) · Psychological Science · doi:10.1111/1467-9280.01433 · Cited by 1,371
Harvesting implicit group attitudes and beliefs from a demonstration web site.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002) · Group Dynamics: Theory, Research, and Practice · doi:10.1037/1089-2699.6.1.101 · Cited by 1,321
Development of personality in early and middle adulthood: Set like plaster or persistent change?
Srivastava, S., John, O. P., Gosling, S. D., & Potter, J. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.84.5.1041 · Cited by 1,290
Self-citationIn defense of external invalidity.
Mook, D. G. (1983) · American Psychologist · doi:10.1037/0003-066x.38.4.379 · Cited by 1,284
Handbook of Research Methods in Social and Personality Psychology: Author Index
Reis, H. T., & Judd, C. M. (2013) · Cambridge University Press eBooks · doi:10.1017/cbo9780511996481 · Cited by 1,080
Global self-esteem across the life span.
Robins, R. W., Trzesniewski, K. H., Tracy, J. L., Gosling, S. D., & Potter, J. (2002) · Psychology and Aging · doi:10.1037/0882-7974.17.3.423 · Cited by 836
Self-citationRichman, W. L., Kiesler, S. B., Weisband, S. P., & Drasgow, F. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.5.754 · Cited by 821
Psychology as a means of promoting human welfare.
Miller, G. A. (1969) · American Psychologist · doi:10.1037/h0028988 · Cited by 679
Handbook of Research Methods. in Social and Personality Psychology
Lai, H. L. T. (2000) · Cited by 629
Individual differences in narcissism: Inflated self-views across the lifespan and around the world
Foster, J. D., Keith Campbell, W., & Twenge, J. M. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00026-6 · Cited by 625
World-Wide Web survey research: Benefits, potential problems, and solutions
Schmidt, W. C. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204826 · Cited by 620
Using the Internet for psychological research: Personality testing on the World Wide Web
Buchanan, T., & Smith, J. L. (1999) · British Journal of Psychology · doi:10.1348/000712699161189 · Cited by 596
Response Effects in the Electronic Survey
Kiesler, S. B., & Sproull, L. S. (1986) · Public Opinion Quarterly · doi:10.1086/268992 · Cited by 568
Reips, U. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50005-8 · Cited by 493
Psychological Experiments on the Internet
(2000) · Elsevier eBooks · doi:10.1016/b978-0-12-099980-4.x5000-x · Cited by 466
e-Perceptions: Personality Impressions Based on Personal Websites.
Vazire, S., & Gosling, S. D. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.87.1.123 · Cited by 460
Self-citationValidity of Web-Based Psychological Research
Krantz, J. H., & Dalal, R. S. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50003-4 · Cited by 429
Goldberg, L. R., Sweeney, D., Merenda, P. F., & Hughes, J. E. (1998) · Personality and Individual Differences · doi:10.1016/s0191-8869(97)00110-4 · Cited by 378
(2000) · Cited by 375
Self-citationPrinciples for Constructing Web Surveys
Dillman, D. A., Tortora, R. D., & Bowker, D. (1998) · Cited by 292
Virtual subjects: Using the Internet as an alternative source of subjects and research environment
Smith, M., & Leigh, B. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210601 · Cited by 290
Jourard, S. M. (1968) · Medical Entomology and Zoology · Cited by 270
An Assessment of the Generalizability of Internet Surveys
Best, S. J., Krueger, B. S., Hubbard, C., & Smith, A. (2001) · Social Science Computer Review · doi:10.1177/089443930101900201 · Cited by 249
The Integrity of Web-Delivered Experiments: Can You Trust the Data?
McGraw, K. O., Tew, M. D., & Williams, J. E. (2000) · Psychological Science · doi:10.1111/1467-9280.00296 · Cited by 190
Potential of the Internet for Personality Research
Buchanan, T. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50006-x · Cited by 175
Proper methodologies for psychological and sociological studies conducted via the Internet
Hewson, C. M., Laurent, D., & Vogel, C. M. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204763 · Cited by 163
A comparison of World-Wide Web and paper-and-pencil personality questionnaires
Pettit, F. A. (2002) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195423 · Cited by 158
Taylor, H. (2000) · International Journal of Market Research · doi:10.1177/147078530004200104 · Cited by 153
Assessment of suicide risk by computer‐delivered self‐rating questionnaire: preliminary findings
Levine, S., Ancill, R. J., & Roberts, A. P. (1989) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1989.tb01330.x · Cited by 112
Self-Selected and Randomly Selected Respondents in a Computer Network Survey
Walsh, J. P., Kiesler, S. B., Sproull, L. S., & Hesse, B. W. (1992) · Public Opinion Quarterly · doi:10.1086/269315 · Cited by 111
The making of a personality inventory: Help from the WWW
Pasveer, K. A., & Ellard, J. H. T. (1998) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200659 · Cited by 103
INFLUENCE OF PLAIN VS. FANCY DESIGN ON RESPONSE RATES FOR WEB SURVEYS
Dillman, D. A., Tortora, R. D., & Conradt, J. (1999) · Cited by 101
Guidelines for Internet Research
Michalak, E. E., & Szabó, A. (1998) · European Psychologist · doi:10.1027/1016-9040.3.1.70 · Cited by 90
Muscle metabolism and PSE pork
BOWKER, B. C., Grant, A. L., Forrest, J. C., & Gerrard, D. E. (2000) · Journal of Animal Science · doi:10.2527/jas.00.079es1001c · Cited by 85
Introduction to Behavioral Research on the Internet
Birnbaum, M. H. (2000) · Medical Entomology and Zoology · Cited by 78
Gosling, S. D., & Bonnenburg, A. V. (1998) · Anthrozoös · doi:10.2752/089279398787000661 · Cited by 74
Self-citationThe political economy of structural adjustment in Tunisia and Algeria
Dillman, B. L. (1998) · The Journal of North African Studies · doi:10.1080/13629389808718335 · Cited by 67
How to Conduct Behavioral Research over the Internet: A Beginner's Guide to HTML and CGI/Perl
Fraley, R. C. (2004) · Medical Entomology and Zoology · Cited by 45
Methodological and Ethical Issues in Conducting Social Psychology Research via the Internet
Birnbaum, M. H. (2008) · Sage eBooks · doi:10.4135/9781412976190.n16 · Cited by 44
CGI scripts: Gateways to World-Wide Web power
Kieley, J. M. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204757 · Cited by 28
Some nondeprivation variables which influence the "effects" of experimental sensory deprivation.
Jackson, C. W., & Pollard, J. C. (1966) · Journal of Abnormal Psychology · doi:10.1037/h0023741 · Cited by 18
CGI scripts: A strategy for between-subjects experimental group assignment on the World-Wide Web
Morrow, R. H., & McKee, A. J. (1998) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200658 · Cited by 16
Issues in Evaluating Laboratory Education
ARGYRISO, C. (1968) · Industrial Relations A Journal of Economy and Society · doi:10.1111/j.1468-232x.1968.tb00465.x · Cited by 12
A Rejoinder to Dunnette and Campbell
Argyris, C. (1968) · Industrial Relations A Journal of Economy and Society · doi:10.1111/j.1468-232x.1968.tb00467.x · Cited by 5
Johnson, G. A. L. (2000) · Geology Today · doi:10.1046/j.1365-2451.2000.1602006.x · Cited by 3
Johnson, D. H., & Johnson, M. P. (2001) · Journal of Glaucoma · doi:10.1097/00061198-200108000-00022 · Cited by 3
Goldberg, A. I. (2003) · Journal of the American Psychoanalytic Association · doi:10.1177/00030651030510030304 · Cited by 2
The antenna test facilities at Da Yeh University
Chang, D.-C., & Chang, K.-C. (2003) · International Symposium on Antennas, Propagation and EM Theory (ISAPE) · doi:10.1109/isape.2003.1276704 · Cited by 2
(2000) · Atomization and Sprays · doi:10.1615/atomizspr.v10.i3-5.10 · Cited by 2
Mendenhall, W. M. (2001) · Hematology/Oncology Clinics of North America · doi:10.1016/s0889-8588(05)70208-2 · Cited by 2
Kraut, R. A. (2003) · Implant Dentistry · doi:10.1097/00008505-200312030-00004 · Cited by 2
Lenhart, V. (2003) · VS Verlag für Sozialwissenschaften eBooks · doi:10.1007/978-3-663-10985-3_7 · Cited by 2
Summary report of journal operations, 2002.
(2003) · American Psychologist · doi:10.1037/0003-066x.58.8.663 · Cited by 2
Research works printed but not linked: 7
- Azar, 2000, A web of research: They're fun, they're fast, and they save money, but do Web experiments yield quality results?, Monitor on Psychology, 31, 42
- 10.1037/0022-3514.75.3.729 · DOI · Paper page
- Dillman. 1998 · DOI
- Locke, 1995, Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats., Journal of Social Behavior and Personality, 10, 255
- 2000, Digest of education statistics 2000.
- Skitka, (in press). Social psychological research and me Internet: The promise and the perils of a new methodological frontier.
- 2000, Population, 1
Cited by
Sorted by citations
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
“To examine how MTurk samples compare with the diversity of standard Internet samples, we compared the demographics of 3,006 MTurk participants with those in a large Internet sample (Gosling et al., 2004).”
Self-citationPositive Psychology Progress: Empirical Validation of Interventions.
Seligman, M. E. P., Steen, T. A., Park, N., & Peterson, C. (2005) · American Psychologist · doi:10.1037/0003-066x.60.5.410 · Cited by 5,947
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
“Based on a comprehensive analysis, for example, Gosling et al. (2004) conclude that Internet samples tend to be diverse, are not adversely affected by nonserious or habitual responders, and produce findings consistent with traditional methods.”
ResultRunning experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
“More importantly, as we demonstrate in Section 4, non-response error seems to be less of a concern in Mechanical Turk samples than in Internet convenience samples recruited through other means.”
BackgroundResultInfluentialDevelopment and validation of the short grit scale (grit-s)
Duckworth, A. L., & Quinn, P. D. (2009) · Journal of Personality Assessment · doi:10.1080/00223890802634290 · Cited by 3,376
“Gosling, Vazire, Srivastava, and John (2004) reviewed evidence that questionnaires completed on the Internet demonstrate psychometric properties that are very similar to those of paper-and-pencil measures.”
Metadata
- Authors
- Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P.
- Affiliations
- The University of Texas at Austin (US)
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- American Psychologist
- Published
- 2004
- Volume
- 59
- Issue
- 2
- Pages
- 93-104
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W1992490637
- Cited by
- 2,932
- References
- 72
- Citations by year
- 2026: 40 · 2025: 64 · 2024: 72 · 2023: 85 · 2022: 123 · 2021: 135 · 2020: 157 · 2019: 163 · 2018: 148 · 2017: 170 · 2016: 181 · 2015: 192 · 2014: 218 · 2013: 184 · 2012: 189
- FWCI
- 261.73
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Psychologist (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Focus Groups and Qualitative Methods
- Keywords
- online data collection, paper-and-pencil questionnaires, sample representativeness, data quality, response format effects, generalizability, self-report questionnaire
- MeSH terms
- Adult, Aged, Attitude, Female, Humans, Male, Middle Aged, Psychology, Surveys and Questionnaires, Evidence-Based Medicine, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
