Conducting perception research over the internet: A tutorial review
Open accessPublished version of a preprint
Abstract
This article provides an overview of the recent literature on the use of internet-based testing to address important questions in perception research. Our goal is to provide a starting point for the perception researcher who is keen on assessing this tool for their own research goals. Internet-based testing has several advantages over in-lab research, including the ability to reach a relatively broad set of participants and to quickly and inexpensively collect large amounts of empirical data, via services such as Amazon's Mechanical Turk or Prolific Academic. In many cases, the quality of online data appears to match that collected in lab research. Generally-speaking, online participants tend to be more representative of the population at large than those recruited for lab based research. There are, though, some important caveats, when it comes to collecting data online. It is obviously much more difficult to control the exact parameters of stimulus presentation (such as display characteristics) with online research. There are also some thorny ethical elements that need to be considered by experimenters. Strengths and weaknesses of the online approach, relative to others, are highlighted, and recommendations made for those researchers who might be thinking about conducting their own studies using this increasingly-popular approach to research in the psychological sciences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Alternative platforms offer access to new, more naïve populations than MTurk's.
“These alternative platforms offer access to new, more naïve populations than MTurk's, and have fewer restrictions on the types of assignments researchers may ask participants to undertake (Vakharia & Lease, 2015;Woods et al., 2015).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Woods et al., 2015
Alternative platforms have fewer restrictions on the types of assignments researchers may ask participants to undertake.
“These alternative platforms offer access to new, more naïve populations than MTurk's, and have fewer restrictions on the types of assignments researchers may ask participants to undertake (Vakharia & Lease, 2015;Woods et al., 2015).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Woods et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,169 · see citing works
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Collier, G. A. (1973) · Language · doi:10.2307/412128 · Cited by 2,840 · see citing works
DMDX: A Windows display program with millisecond accuracy
Forster, K. I., & Forster, J. C. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195503 · Cited by 2,775 · see citing works
A wandering mind is an unhappy mind
Killingsworth, M. A., & Gilbert, D. T. (2010) · Science · doi:10.1126/science.1192439 · Cited by 2,575 · see citing works
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Wechsler Adult Intelligence Scale--Fourth Edition
Wechsler, D. (2012) · PsycTESTS Dataset · doi:10.1037/t15169-000 · Cited by 2,232 · see citing works
Basic Color Terms: Their Universality and Evolution.
Morrill, W. T., Berlin, B., & Kay, P. (1971) · Man · doi:10.2307/2798490 · Cited by 2,182 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,783 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Wechsler Adult Intelligence Scale—Fourth Edition (WAIS-IV)
Lani, J. A., & Wechsler, D. (2010) · Cited by 1,686 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
2014 IEEE Symposium on Security and Privacy
(2014) · doi:10.1109/sp16517.2014 · Cited by 1,388 · see citing works
De-anonymizing Social Networks
Narayanan, A., & Shmatikov, V. (2009) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2009.22 · Cited by 1,324 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
A Review of Facebook Research in the Social Sciences
Wilson, R. E., Gosling, S. D., & Graham, L. T. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612442904 · Cited by 1,226 · see citing works
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Self-citationThe viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works
Duchaine, B., & Nakayama, K. (2005) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2005.07.001 · Cited by 1,040 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Kraut, R. E., Olson, J., Banaji, M. R., Bruckman, A. S., Cohen, J., & Couper, M. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.105 · Cited by 922 · see citing works
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 890 · see citing works
Learning and memorization of classifications.
Shepard, R. N., Hovland, C. I., & Jenkins, H. M. (1961) · The Psychological Monographs · doi:10.1037/h0093825 · Cited by 824 · see citing works
Rule of Law and Constitutional Democracy
(2015) · DergiPark (Istanbul University) · doi:10.2139/ssrn · Cited by 804 · see citing works
Intuitive Cooperation and the Social Heuristics Hypothesis: Evidence from 15 Time Constraint Studies
Rand, D. G., Peysakhovich, A., Kraft‐Todd, G., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 730 · see citing works
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670 · see citing works
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · see citing works
Visual illusion induced by sound
Shams, L., Kamitani, Y., & Shimojo, S. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00069-1 · Cited by 668 · see citing works
Shapiro, K. L., Raymond, J. E., & Arnell, K. M. (1997) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(97)01094-2 · Cited by 654 · see citing works
The Smartphone Psychology Manifesto
Miller, G. F. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441215 · Cited by 618 · see citing works
Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence.
Eimer, M., & Schlaghecken, F. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.24.6.1737 · Cited by 467 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463 · see citing works
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458 · see citing works
Friendship prediction and homophily in social media
Aiello, L., Barrat, A., Schifanella, R., Cattuto, C., Markines, B., & Menczer, F. (2012) · TWEB · doi:10.1145/2180861.2180866 · Cited by 447 · see citing works
Are your participants gaming the system?
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 422 · see citing works
Eriksen, C. W. (1995) · Visual Cognition · doi:10.1080/13506289508401726 · Cited by 412 · see citing works
Ensuring the data-rich future of the social sciences
King, G. (2011) · Science · doi:10.1126/science.1197872 · Cited by 394 · see citing works
Music, Cognition, Culture, and Evolution
Cross, I. R. M. (2001) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2001.tb05723.x · Cited by 385 · see citing works
Resolving the question of color naming universals
Kay, P., & Regier, T. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1532837100 · Cited by 329 · see citing works
Ethics guidelines for internet-mediated research
(2021) · British Psychological Society eBooks · doi:10.53841/bpsrep.2021.rep155.5 · Cited by 323 · see citing works
Dimensions of Internet science
Reips, U., & Bošnjak, M. (2001) · Cited by 319 · see citing works
Nosofsky, R. M., Gluck, M. A., Palmeri, T. J., McKinley, S. C., & Glauthier, P. (1994) · Memory & Cognition · doi:10.3758/bf03200862 · Cited by 292 · see citing works
Mind-wandering and negative mood: Does one thing really lead to another?
Poerio, G. L., Totterdell, P., & Miles, E. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2013.09.012 · Cited by 253 · see citing works
Audiovisual multisensory integration
Spence, C. J. (2007) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.28.61 · Cited by 252 · see citing works
Self-citationMarx, D. M., & Goff, P. A. (2005) · British Journal of Social Psychology · doi:10.1348/014466604x17948 · Cited by 245 · see citing works
de Leeuw, J. R., & Motz, B. A. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0567-2 · Cited by 242 · see citing works
Nieuwenhuis, S., Stins, J. F., Posthuma, D., Polderman, T. J. C., Boomsma, D. I., & de Geus, E. J. C. (2006) · Memory & Cognition · doi:10.3758/bf03193270 · Cited by 214 · see citing works
Luca, M. D., Machulla, T., & Ernst, M. O. (2009) · Journal of Vision · doi:10.1167/9.12.7 · Cited by 209 · see citing works
Hedonic mediation of the crossmodal correspondence between taste and shape
Velasco, C., Woods, A. T., Deroy, O. D., & Spence, C. J. (2014) · Food Quality and Preference · doi:10.1016/j.foodqual.2014.11.010 · Cited by 181 · see citing works
Self-citationDufau, S., Duñabeitia, J. A., Moret‐Tatay, C., McGonigal, A., Peeters, D., Alario, F. ‐., Balota, D. A., Brysbaert, M., Carreiras, M., Ferrand, L., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024974 · Cited by 177 · see citing works
Emotional reactions to losing explain gender differences in entering a risky lottery
Eriksson, K., & Simpson, B. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500001030 · Cited by 171 · see citing works
Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments
Reimers, S., & Stewart, N. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0471-1 · Cited by 171 · see citing works
Cross-cultural differences in crossmodal correspondences between basic tastes and visual features
Wan, X. I., Woods, A. T., van den Bosch, J. J. F., McKenzie, K. J., Velasco, C., & Spence, C. J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01365 · Cited by 169 · see citing works
Self-citationBrief body-scan meditation practice improves somatosensory perceptual decision making
Mirams, L., Poliakoff, E., Brown, R. J., & Lloyd, D. M. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2012.07.009 · Cited by 157 · see citing works
Incentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146 · see citing works
That Sounds Sweet: Using Cross‐Modal Correspondences to Communicate Gustatory Attributes
Knoeferle, K. M., Woods, A., Käppler, F., & Spence, C. (2015) · Psychology & Marketing · doi:10.1002/mar.20766 · Cited by 137 · see citing works
Crowdsourcing for Cognitive Science – The Utility of Smartphones
Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100662 · Cited by 129 · see citing works
Tracing the Trajectory of Skill Learning With a Very Large Sample of Online Game Players
Stafford, T., & Dewar, M. (2013) · Psychological Science · doi:10.1177/0956797613511466 · Cited by 111 · see citing works
Imagery paradigms: How vulnerable are they to experimenters' expectations?
Intons-Peterson, M. J. (1983) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.9.3.394 · Cited by 109 · see citing works
Collecting response times using Amazon Mechanical Turk and Adobe Flash
Simcox, T., & Fiez, J. A. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0345-y · Cited by 97 · see citing works
Red Hot: The Crossmodal Effect of Color Intensity on Perceived Piquancy
Shermer, D. Z., & Levitan, C. (2014) · Multisensory Research · doi:10.1163/22134808-00002457 · Cited by 96 · see citing works
Self-citationPlant, R. R., & Turner, G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.3.598 · Cited by 96 · see citing works
Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities
Reimers, S., & Stewart, N. (2007) · Behavior Research Methods · doi:10.3758/bf03193004 · Cited by 93 · see citing works
Cross-Cultural Color-Odor Associations
Levitan, C., Ren, J., Woods, A. T., Boesveldt, S., Chan, J., McKenzie, K. J., Dodson, M., Levin, J., Leong, C. X. R., & van den Bosch, J. J. F. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101651 · Cited by 90 · see citing works
Self-citationVelasco, C., Wan, X. I., Montejo, A. S., Woods, A. T., Oñate, G. A., Mu, B., & Spence, C. J. (2014) · Food Quality and Preference · doi:10.1016/j.foodqual.2014.05.011 · Cited by 85 · see citing works
Self-citationAnonymizing Health Data: Case Studies and Methods to Get You Started
Emam, K. E., & Arbuckle, L. (2013) · O'Reilly Media, Inc. eBooks · Cited by 85 · see citing works
Exploring implicit and explicit crossmodal colour–flavour correspondences in product packaging
Piqueras‐Fiszman, B., Velasco, C., & Spence, C. J. (2012) · Food Quality and Preference · doi:10.1016/j.foodqual.2012.02.010 · Cited by 84 · see citing works
Self-citationMechanical Turk is Not AnonymousPreprint
Lease, M., Hullman, J. R., Bigham, J. P., Bernstein, M. S., Kim, J., Lasecki, W. S., Bakhshi, S., Mitra, T., & Miller, R. C. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2228728 · Cited by 82 · see citing works
Velasco, C., Wan, X. I., Knoeferle, K. M., Zhou, X., Montejo, A. S., & Spence, C. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00301 · Cited by 77 · see citing works
Self-citationScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition
Schubert, T. W., Murteira, C., Collins, E. C., & Lopes, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067769 · Cited by 71 · see citing works
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J., Paolacci, G., & Mueller, P. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_30 · Cited by 70 · see citing works
Gray colored glasses: Is major depression partially a sensory perceptual disorder?
Fitzgerald, P. J. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.06.045 · Cited by 69 · see citing works
Wan, X. I., Woods, A. T., Seoul, K.-H., Butcher, N. L., & Spence, C. J. (2014) · Food Quality and Preference · doi:10.1016/j.foodqual.2014.07.004 · Cited by 67 · see citing works
Self-citationWan, X. I., Velasco, C., Michel, C., Mu, B., Woods, A. T., & Spence, C. J. (2014) · Flavour · doi:10.1186/2044-7248-3-3 · Cited by 67 · see citing works
Self-citationExperimenter sex effects in behavioral research.
Rumenik, D. K., Capasso, D. R., & Hendrick, C. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.5.852 · Cited by 60 · see citing works
Introduction to the Special Section: Data, Data, Everywhere . . . Especially in My File Drawer
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432124 · Cited by 58 · see citing works
Response time accuracy in Apple Macintosh computers
Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0069-9 · Cited by 57 · see citing works
Garaizar, P., Vadillo, M. A., López–de–Ipiña, D., & Matute, H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0085108 · Cited by 52 · see citing works
Woods, A. T., Spence, C. J., Butcher, N. L., & Deroy, O. D. (2013) · i-Perception · doi:10.1068/i0586 · Cited by 46 · see citing works
Self-citationPéer, E., Samat, S., Brandimarte, L., & Acquisti, A. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2594183 · Cited by 44 · see citing works
Nicholls, M. E. R., Loveless, K. M., Thomas, N. A., Loetscher, T., & Churches, O. F. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.925481 · Cited by 44 · see citing works
Psychophysical contrast calibration
To, L., Woods, R. L., Goldstein, R. B., et al. (2013) · Vision Research · doi:10.1016/j.visres.2013.04.011 · Cited by 41 · see citing works
Sirota, M., Kostovičová, L., & Vallée‐Tourangeau, F. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.972420 · Cited by 39 · see citing works
Brand, A., & Bradley, M. T. (2011) · Social Science Computer Review · doi:10.1177/0894439311415604 · Cited by 39 · see citing works
Assessing the Role of Taste Intensity and Hedonics in Taste–Shape Correspondences
Velasco, C., Woods, A. T., Liu, J., & Spence, C. J. (2015) · Multisensory Research · doi:10.1163/22134808-00002489 · Cited by 38 · see citing works
Self-citationMartire, K., & Watkins, I. (2015) · Science & justice : journal of the Forensic Science Society · doi:10.1016/j.scijus.2015.01.002 · Cited by 34 · see citing works
A systematic review of tools, languages, and methodologies for mashup development
Paredes‐Valverde, M. A., Alor‐Hernández, G., Rodríguez‐González, A., et al. (2013) · Software Practice and Experience · doi:10.1002/spe.2233 · Cited by 28 · see citing works
Methods to Test Visual Attention Online
Yung, A., Cardoso-Leite, P., Dale, G., Bavelier, D., & Green, C. S. (2015) · Journal of Visualized Experiments · doi:10.3791/52470 · Cited by 26 · see citing works
Ethical issues in online research
James, N., & Busher, H. (2015) · Educational Research and Evaluation · doi:10.1080/13803611.2015.1024420 · Cited by 23 · see citing works
Testing the viability of webDMDX for masked priming experiments
Witzel, J., Cornelius, S., Witzel, N., et al. (2013) · The Mental Lexicon · doi:10.1075/ml.8.3.07wit · Cited by 21 · see citing works
Gibraltar: exposing hardware devices to web pages using AJAX
Lin, K., Chu, D. C. Y., Mickens, J. W., et al. (2012) · Cited by 11 · see citing works
Bauer, B. (2015) · Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale · doi:10.1037/cep0000043 · Cited by 11 · see citing works
Breaking Fresh Ground in Human–Media Interaction Research
Nijholt, A. (2014) · Frontiers in ICT · doi:10.3389/fict.2014.00004 · Cited by 10 · see citing works
Do research ethics need updating for the digital age?
Ross, M. W. (2014) · PsycEXTRA Dataset · doi:10.1037/e577942014-005 · Cited by 8 · see citing works
A Preliminary Study of Daily Sample Composition on Amazon Mechanical TurkPreprint
Lakkaraju, K. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2560840 · Cited by 4 · see citing works
Bouba Kiki et al over several samples from different populations
Woods, A. T., Levitan, C., Natlie, B., et al. (2015) · Figshare · doi:10.6084/m9.figshare.1408803 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Thompson S Internet connectivity: addiction and dependency study Doctoral dissertation 1996 Available at http://www.academia.edu/download/30519620/Thompson_-_IADS.pdf
Cited by
Sorted by citations
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Leo, X., & Huh, Y. E. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106520 · Cited by 261 · see citing works
Spence, C., & Velasco, C. (2018) · Food Quality and Preference · doi:10.1016/j.foodqual.2018.03.008 · Cited by 177 · see citing works
Self-citationChen, J., Mishler, S., & Hu, B. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3051137 · Cited by 40 · see citing works
Woods, A. T., Marmolejo-Ramos, F., Velasco, C., & Spence, C. (2016) · i-Perception · doi:10.1177/2041669516663750 · Cited by 37 · see citing works
Self-citation
- Authors
- Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C.
- Affiliations
- University of Oxford (GB)
- Occidental College (US)
- Tsinghua University (CN)
- Venue
- PeerJ
- Publisher
- PeerJ
- Published
- 2015-07-23
- Volume
- 3
- Pages
- e1058
- ISSN
- 2167-8359 (ISSN-L)
- PMC
- PMC4517966
- OpenAlex
- W1524842461
- Cited by
- 301
- References
- 106
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 16 · 2023: 26 · 2022: 44 · 2021: 36 · 2020: 32 · 2019: 51 · 2018: 27 · 2017: 14 · 2016: 29 · 2015: 8
- FWCI
- 38.92
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- PeerJ (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Arts and Humanities Research Council (AH/L007053/1)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- web-based testing, Amazon Mechanical Turk, participant recruitment, research ethics, large-scale data collection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
