Validating Internet research: A test of the psychometric equivalence of Internet and in-person samples
Open accessAbstract
This study evaluated the psychometric equivalency of Web-based research. The Sexual Boredom Scale was presented via the World-Wide Web along with five additional scales used to validate it. A subset of 533 participants that matched a previously published sample (Watt & Ewing, 1996) on age, gender, and race was identified. An 8 x 8 correlation matrix from the matched Internet sample was compared via structural equation modeling with a similar 8 x 8 correlation matrix from the previously published study. The Internet and previously published samples were psychometrically equivalent. Coefficient alpha values calculated on the matched Internet sample yielded reliability coefficients almost identical to those for the previously published sample. Factors such as computer administration and uncontrollable administration settings did not appear to affect the results. Demographic data indicated an overrepresentation of males by about 6% and Caucasians by about 13% relative to the U.S. Census (2000). A total of 2,230 participants were obtained in about 8 months without remuneration. These results suggest that data collection on the Web is (1) reliable, (2) valid, (3) reasonably representative, (4) cost effective, and (5) efficient.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Web-based testing methods and traditional testing procedures provide similar results.
“This provides additional evidence that Web-based testing methods and traditional testing procedures provide similar results (cf. Germine et al., 2012; Meyerson & Tryon, 2003).”
From When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span (2015) · cites this paper as Meyerson & Tryon, 2003
Web data can be as reliable as data collected in the lab and/or using traditional methods.
“Systematic comparisons between data collected from lab vs. Web-based samples have demonstrated that Web data can be as reliable as data collected in the lab and/or using traditional methods (Germine, Nakayama, Duchaine, Chabris, Chatterjee & Wilmer, 2012; Meyerson & Tryon, 2003).”
From When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span (2015) · cites this paper as Meyerson & Tryon, 2003
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Nolen-Hoeksema, S., & Morrow, J. (1991) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.61.1.115 · Cited by 3,276 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Bailey, R. D., Foote, W. E., & Throckmorton, B. (2000). Human sexual behavior: A comparison of college and Internet surveys. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 141-168). San Diego: Academic Press.
- Batinic, B., Reips, U.-D., & Bosnjak, M. (Eds.) (2002). Online social sciences. Cambridge, MA: Hogrefe & Huber.
- Birnbaum, M. H. (Ed.) (2000a). Introduction to behavioral research on the Internet. Upper Saddle River, NJ: Prentice-Hall.
- Birnbaum, M. H. (2000b). Psychological experiments on the Internet. San Diego: Academic Press.
- Browne, M. W. (1993). The use of causal indicators in covariance structure models: Some practical issues. Psychological Bulletin, 114, 533-541.
- Buchanan, T., & Smith, J. L. (1999b). Using the Internet for psycho- logical research: Personality testing on the World-Wide Web. British Journal of Psychology, 90, 125-144.
- Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71-75.
- Farmer, R., & Sundberg, N. D. (1986). Boredom proneness-The de- velopment and correlates of a new scale. Journal of Personality As- sessment, 50, 4-17.
- Gartner says average U.S. Inter- net user is 41 years old with an income of $65,000 [Press release]. Stamford, CT: Author. Retrieved January 15, 2001, from http:// www.gartner.com/5_about/press_room/pr20001030a.html.
- Ghiselli, E. E. (1964). Theory of psychological measurement. New York: McGraw-Hill.
- Honaker, L. M. (1988). The equivalency of computerized and con- ventional MMPI administration: A critical review. Clinical Psychol- ogy Review, 8, 561-577.
- Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psycholog- ical research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35-60). San Diego: Academic Press.
- Snell, W. E., & Papini, D. R. (1989). The sexuality scale: An instru- ment to measure sexual-esteem, sexual-depression, and sexual- preoccupation. Journal of Sex Research, 26, 256-263.
- Snyder, M. (1974). Self-monitoring of expressive behavior. Journal of Personality & Social Psychology, 30, 526-537.
- Snyder, M., Berscheid, E., & Glick, P. (1985). Focusing on the exte- rior and the interior: Two investigations of the initiation of personal relationships. Journal of Personality & Social Psychology, 48, 1427- 1439.
- Snyder, M., & Gangestad, S. W. (1986). On the nature of self- monitoring: Matters of assessment, matters of validity. Journal of Personality & Social Psychology, 51, 125-139.
- Watt, J. D., & Ewing, J. E. (1996). Toward the development and vali- dation of a measure of sexual boredom. Journal of Sex Research, 33, 57-66.
Cited by
Sorted by citations
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Hartshorne, J. K., & Germine, L. T. (2015) · Psychological Science · doi:10.1177/0956797614567339 · Cited by 788 · see citing works
Lowry, P. B., D’Arcy, J., Hammer, B., & Moody, G. D. (2016) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2016.06.002 · Cited by 301 · see citing works
Tully, P. J., Zajac, I. T., & Venning, A. J. (2009) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-009-9306-4 · Cited by 177 · see citing works
Høifødt, R. S., Lillevoll, K. R., Griffiths, K. M., Wilsgaard, T., Eisemann, M., Waterloo, K., & Kolstrup, N. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2714 · Cited by 154 · see citing works
“Research indicates that Internet data on this measure is as reliable and valid as paper-and-pencil data [81].”
Moritz, S., Jelinek, L., Hauschildt, M., & Naber, D. (2010) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2010.12.2/smoritz · Cited by 113 · see citing works
The appropriated racial oppression scale: Development and preliminary validation
Campón, R. R., & Carter, R. T. (2015) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/cdp0000037 · Cited by 68 · see citing works
Moritz, S., Fricke, S., Treszl, A., & Wittekind, C. E. (2012) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2011.11.001 · Cited by 48 · see citing works
Briones, E. M., & Benham, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0710-8 · Cited by 48 · see citing works
van Gastel, W. A., Vreeker, A., Schubart, C. D., MacCabe, J. H., Kahn, R. S., & Boks, M. P. M. (2014) · Schizophrenia Research · doi:10.1016/j.schres.2014.04.023 · Cited by 37 · see citing works
- Authors
- Meyerson, P., & Tryon, W. W.
- Affiliations
- Fordham University (US)
- Corresponding author
- Tryon, Warren W.
- ORCID
- Tryon, Warren W.
- Venue
- Behavior Research Methods, Instruments, & Computers
- Published
- 2003-11
- Volume
- 35
- Issue
- 4
- Pages
- 614-620
- ISSN
- 0743-3808 (ISSN-L)
- 1532-5970
- OpenAlex
- W1966375368
- Cited by
- 197
- References
- 1
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 7 · 2021: 6 · 2020: 12 · 2019: 11 · 2018: 7 · 2017: 7 · 2016: 7 · 2015: 13 · 2014: 15 · 2013: 14 · 2012: 18
- FWCI
- 12.44
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Behavior Research Methods, Instruments, & Computers (publishedVersion)
- PubMed (publishedVersion)
- Fordham Research Commons (Fordham University) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- online research, online data collection, psychometric equivalence, scale validation, reliability statistics, sample representativeness, matched samples, structural equation modeling
- MeSH terms
- Adult, Boredom, Female, Humans, Male, Psychometrics, Sexual Behavior, Internet, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
