Standards for Internet-Based Experimenting

Abstract

This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of motivational confounding, and pre-testing. Several techniques are explained, including “warm-up,” “high hurdle,” password methods, “multiple site entry,” randomization, and the use of incentives. The article concludes by proposing sixteen standards for Internet-based experimenting.

Cites (35)

1. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.
   Orne, Martin T. · 1962 · American Psychologist
2. Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity
   Joinson, Adam N. · 2001 · European Journal of Social Psychology
3. Can You See the Real Me? Activation and Expression of the “True Self” on the Internet
   Bargh, John A., McKenna, Katelyn Y. A., Fitzsimons, Grainne M. · 2002 · Journal of Social Issues
4. Experimenter Effects in Behavioral Research.
   Titchener, James L. · 1967 · Archives of Internal Medicine
5. World-Wide Web survey research: Benefits, potential problems, and solutions
   Schmidt, William C. · 1997 · Behavior Research Methods, Instruments, & Computers
6. Response Effects in the Electronic Survey
   Kiesler, Sara, Sproull, Lee S. · 1986 · Public Opinion Quarterly
7. Social Influence in Computer-Mediated Communication: The Effects of Anonymity on Group Behavior
   Postmes, Tom, Spears, Russell, Sakhel, Khaled, de Groot, Daphne · 2001 · Personality and Social Psychology Bulletin
8. The Web Experiment Method
   Reips, Ulf-Dietrich · 2000 · Psychological Experiments on the Internet
9. Social Science Computer Review
   1987 · Social Science Microcomputer Review
10. Validity of Web-Based Psychological Research
    Krantz, John H., Dalal, Reeshad · 2000 · Psychological Experiments on the Internet

Show all 35 references

Cited by (37)

1. Conducting behavioral research on Amazon’s Mechanical Turk
   Mason, Winter, Suri, Siddharth · 2012 · Behavior Research Methods
2. Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
   Crump, Matthew J. C., McDonnell, John V., Gureckis, Todd M. · 2013 · PLoS ONE
3. The world of emotions is not two-dimensional
   Fontaine, Johnny R.J., Scherer, Klaus R., Roesch, Etienne B., Ellsworth, Phoebe C. · 2007 · Psychological Science
4. Conducting Clinical Research Using Crowdsourced Convenience Samples
   Chandler, Jesse, Shapiro, Danielle · 2016 · Annual Review of Clinical Psychology
5. The viability of crowdsourcing for survey research
   Behrend, Tara S., Sharek, David J., Meade, Adam W., Wiebe, Eric N. · 2011 · Behavior Research Methods

See all 554