Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments
Abstract
With the increasing sophistication and ubiquity of the Internet, behavioral research is on the cusp of a revolution that will do for population sampling what the computer did for stimulus control and measurement. It remains a common assumption, however, that data from self-selected Web samples must involve a trade-off between participant numbers and data quality. Concerns about data quality are heightened for performance-based cognitive and perceptual measures, particularly those that are timed or that involve complex stimuli. In experiments run with uncompensated, anonymous participants whose motivation for participation is unknown, reduced conscientiousness or lack of focus could produce results that would be difficult to interpret due to decreased overall performance, increased variability of performance, or increased measurement noise. Here, we addressed the question of data quality across a range of cognitive and perceptual tests. For three key performance metrics-mean performance, performance variance, and internal reliability-the results from self-selected Web samples did not differ systematically from those obtained from traditionally recruited and/or lab-tested samples. These findings demonstrate that collecting data from uncompensated, anonymous, unsupervised, self-selected participants need not reduce data quality, even for demanding cognitive and perceptual experiments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Web-based testing methods and traditional testing procedures provide similar results.
“This provides additional evidence that Web-based testing methods and traditional testing procedures provide similar results (cf. Germine et al., 2012; Meyerson & Tryon, 2003).”
From When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span (2015) · cites this paper as Germine et al., 2012
The convergence adds to the growing body of work indicating that Web-based data is highly reliable.
“This convergence adds to the growing body of work indicating that Web-based data is highly reliable (e.g., Germine et al., 2012).”
From When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span (2015) · cites this paper as Germine et al., 2012
Web data can be as reliable as data collected in the lab and/or using traditional methods.
“Systematic comparisons between data collected from lab vs. Web-based samples have demonstrated that Web data can be as reliable as data collected in the lab and/or using traditional methods (Germine, Nakayama, Duchaine, Chabris, Chatterjee & Wilmer, 2012; Meyerson & Tryon, 2003).”
From When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span (2015) · cites this paper as Germine, Nakayama, Duchaine, Chabris, Chatterjee & Wilmer, 2012
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
Baron‐Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001) · Journal of Child Psychology and Psychiatry · doi:10.1111/1469-7610.00715 · Cited by 4,555 · cited 3× · see citing works
“The tests included are the Cambridge Face Memory Test (CFMT; Duchaine & Nakayama, 2006), the Reading the Mind in the Eyes Test (RMIE; Baron-Cohen et al., 2001), the Verbal Paired Associates Memory Test (VPAM; Wilmer et al., 2010), the Abstract Art Memory Test (AAM; Wilmer et al., 2010) , and the…”
BackgroundMethodGosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · see citing works
Implementing a five-factor personality inventory for use on the internet
Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005) · European Journal of Psychological Assessment · doi:10.1027/1015-5759.21.2.115 · Cited by 301 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Birnbaum, M. H. (2000). Decision making in the lab and on the Web. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 3 -34). San Diego, CA: Academic Press.
- Bowles, D. C., McKone, E., Dawel, A., Duchaine, B., Palermo, R., Schmalzl, L., et al. (2009). Diagnosing prosopagnosia: Effects of ageing, sex, and participant-stimulus ethnic match on the Cambridge Face Memory Test and Cambridge Face Perception Test. Cognitive Neuropsychology, 26, 423 -455.
- Buchanan, T. (2007). Personality testing on the Internet: What we know, and what we do not. In A. N. Joinson, K. Y . McKenna, T. Postmes, &U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 447 -460). New York, NY: Oxford University Press.
- Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125 -144.
- Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155 -159.
- Duchaine, B., & Nakayama, K. (2006). The Cambridge Face Memory Test: Results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants. Neuropsychologia, 44, 576 -585.
- Herzmann, G., Danthiir, V., Schacht, A., Sommer, W., & Wilhelm, O. (2008). Toward a comprehensive test battery for face cognition: Assessment of the tasks. Behavior Research Methods, 40, 840 -857. doi:10.3758/BRM.40.3.840 · DOI
- Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35 -60). San Diego, CA: Academic Press.
- McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of Web-delivered experiments: Can you trust the data? Psychological Science, 11, 502 -506.
- Plomin, R., DeFries, J. C., McClearn, G. E., & McGuffin, P. (2008). Behavioral genetics (5th ed.). New York, NY: W. H. Freeman.
- Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89 -117). San Diego, CA: Academic Press.
- Reips, U.-D. (2007). The methodology of Internet-based experiments. In A. N. Joinson, K. Y. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 373 -390). New York, NY: Oxford University Press.
- Ronalds, G. A., DeStavola, B. L., & Leon, D. A. (2005). The cognitive cost of being a twin: Evidence from comparisons within families in the Aberdeen children of the 1950s cohort study. British Medical Journal, 331, 1306.
- Soto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2011). Age differences in personality traits from 10 to 65: Big Five domains and facets in a large cross-sectional sample. Journal of Personality and Social Psychology, 100, 330 -348.
- Wechsler, D. (2008). Wechsler Adult Intelligence Scale (4th ed.). San Antonio, TX: Pearson.
- Wilmer, J. B., Germine, L., Chabris, C. F., Chatterjee, G., Williams, M., Loken, E., et al. (2010). Human face recognition ability is specific and highly heritable. Proceedings of the National Academy of Sciences, 107, 5238 -5241.
Cited by
Sorted by citations
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
Hartshorne, J. K., & Germine, L. T. (2015) · Psychological Science · doi:10.1177/0956797614567339 · Cited by 788 · see citing works
Self-citationThe Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630 · see citing works
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301 · see citing works
Digital neuropsychology: challenges and opportunities at the intersection of science and software
Germine, L., Reinecke, K., & Chaytor, N. S. (2019) · The Clinical Neuropsychologist · doi:10.1080/13854046.2018.1535662 · Cited by 171 · see citing works
Self-citationQRTEngine: An easy solution for running online reaction time experiments using Qualtrics
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0530-7 · Cited by 166 · see citing works
Buchanan, T. (2016) · Psychological Assessment · doi:10.1037/pas0000192 · Cited by 134 · see citing works
Kornell, N. (2014) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0033699 · Cited by 88 · see citing works
The role of strategy use in working memory training outcomes
Fellman, D., Jylkkä, J., Waris, O., Soveri, A., Ritakallio, L., Haga, S., Salmi, J., Nyman, T. J., & Laine, M. (2020) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104064 · Cited by 64 · see citing works
The Influence of Personality Traits and Cognitive Load on the Use of Adaptive User Interfaces
Gajos, K. Z., & Chauncey, K. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025192 · Cited by 57 · see citing works
Briones, E. M., & Benham, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0710-8 · Cited by 48 · see citing works
A Large Inclusive Study of Human Listening Rates
Bragg, D., Bennett, C., Reinecke, K., & Ladner, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174018 · Cited by 43 · see citing works
Testing enhances learning across a range of episodic memory abilities
Pan, S. C., Pashler, H., Potter, Z. E., & Rickard, T. C. (2015) · Journal of Memory and Language · doi:10.1016/j.jml.2015.04.001 · Cited by 41 · see citing works
Collins, C. L., Pina, A., Carrillo, A., Ghil, E., Smith-Peirce, R. N., Gomez, M., Okolo, P., Chen, Y., Pahor, A., Jaeggi, S. M., et al. (2022) · Journal of Cognitive Enhancement · doi:10.1007/s41465-022-00240-z · Cited by 15 · see citing works
Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance
García García, P., Costanza, E., Verame, J., Nowacka, D., & Ramchurn, S. D. (2021) · Human–Computer Interaction · doi:10.1080/07370024.2018.1453815 · Cited by 17 · see citing works
Poulton, A., Chen, L. P. E., Dali, G., Fox, M., & Hester, R. (2022) · Journal of Medical Internet Research · doi:10.2196/32922 · Cited by 11 · see citing works
“While there has been a substantial increase in the popularity of using the internet as a medium for conducting research in social psychology—which appears to have resulted in larger samples and increased statistical power [26]—this trend has been less evident in the cognitive arena, possibly owing to validity and reli…”
Background
- Authors
- Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B.
- Affiliations
- Harvard University (US)
- Union College (US)
- Wellesley College (US)
- Corresponding author
- Germine, Laura
- Venue
- Psychonomic Bulletin & Review
- Published
- 2012-07-25
- Volume
- 19
- Issue
- 5
- Pages
- 847-857
- ISSN
- 1069-9384 (ISSN-L)
- 1531-5320
- OpenAlex
- W2170263696
- Cited by
- 669
- References
- 4
- Citations by year
- 2026: 27 · 2025: 57 · 2024: 45 · 2023: 64 · 2022: 59 · 2021: 64 · 2020: 62 · 2019: 44 · 2018: 57 · 2017: 46 · 2016: 39 · 2015: 56 · 2014: 28 · 2013: 17 · 2012: 3
- FWCI
- 50.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Psychonomic Bulletin & Review (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Eye Institute (R01 EY01363)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Keywords
- online experiments, online data collection, data quality, cognitive performance, perceptual performance, performance variance, internal consistency, measurement noise, participant motivation
- MeSH terms
- Adolescent, Adult, Age Factors, Aged, Cognition, Face, Female, Humans, Laboratories, Male, Middle Aged, Motivation, Pattern Recognition, Visual, Research Design, Sex Factors, Reproducibility of Results, Patient Selection, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
