The DOI matches its OpenAlex record; 47 of the 47 references it cites were found in a registry or an index.
Standards for Internet-Based Experimenting
Open accessAbstract
This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of motivational confounding, and pre-testing. Several techniques are explained, including "warm-up," "high hurdle," password methods, "multiple site entry," randomization, and the use of incentives. The article concludes by proposing sixteen standards for Internet-based experimenting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,621
Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity
Joinson, A. N. (2001) · European Journal of Social Psychology · doi:10.1002/ejsp.36 · Cited by 1,458
Can You See the Real Me? Activation and Expression of the “True Self” on the Internet
Bargh, J. A., McKenna, K. Y. A., & Fitzsimons, G. M. (2002) · Journal of Social Issues · doi:10.1111/1540-4560.00247 · Cited by 1,300
Experimenter Effects in Behavioral Research.
Titchener, J. L. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00300050109027 · Cited by 1,194
Artifact in behavioral research
Rosenthal, R., & Rosnow, R. L. (1969) · Medical Entomology and Zoology · Cited by 1,186
World-Wide Web survey research: Benefits, potential problems, and solutions
Schmidt, W. C. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204826 · Cited by 620
Response Effects in the Electronic Survey
Kiesler, S., & Sproull, L. S. (1986) · Public Opinion Quarterly · doi:10.1086/268992 · Cited by 566
Social Influence in Computer-Mediated Communication: The Effects of Anonymity on Group Behavior
Postmes, T., Spears, R., Sakhel, K., & de Groot, D. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/01461672012710001 · Cited by 489
Psychological Experiments on the Internet
(2000) · Elsevier eBooks · doi:10.1016/b978-0-12-099980-4.x5000-x · Cited by 463
Reips, U.-D. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50005-8 · Cited by 460
Self-citationValidity of Web-Based Psychological Research
Krantz, J. H., & Dalal, R. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50003-4 · Cited by 422
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384
Dimensions of Internet science
Reips, U., & Bošnjak, M. (2001) · Cited by 319
Self-citationUsing the Internet for Survey Research
Coomber, R. (1997) · Sociological Research Online · doi:10.5153/sro.73 · Cited by 302
On belief bias in syllogistic reasoning.
Klauer, K. C., Musch, J., & Naumer, B. (2000) · Psychological Review · doi:10.1037//0033-295x.107.4.852 · Cited by 283
Batinic, B., Reips, U., & Bošnjak, M. (2002) · Cited by 246
Self-citationA Brief History of Web Experimenting
Musch, J., & Reips, U.-D. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50004-6 · Cited by 229
Self-citationOnline assessment: Desirable or dangerous?
Buchanan, T. (2002) · Professional Psychology: Research and Practice · doi:10.1037//0735-7028.33.2.148 · Cited by 210
Flick, U. (1998) · Cambridge University Press eBooks · Cited by 196
Proper methodologies for psychological and sociological studies conducted via the Internet
Hewson, C. M., Laurent, D., & Vogel, C. M. (1996) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204763 · Cited by 168
The Relevance of Laboratory Studies to Practical Situations
Chapanis, A. (1967) · Ergonomics · doi:10.1080/00140136708930910 · Cited by 163
Internet-Based Psychological Experimenting: Five Dos and Five Don’ts
Reips, U. (2002) · Social Science Computer Review · doi:10.1177/08939302020003002 · Cited by 135
Self-citationFinancial incentives, personal information and drop-out in online studies
Frick, A., Bächtiger, M.-T., & Reips, U. (2001) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-19758 · Cited by 128
Self-citationSubject selection bias in psychological research.
Smart, R. G. (1966) · The Canadian Psychologist · doi:10.1037/h0083096 · Cited by 126
Differential effects of Ecstasy and cannabis on self‐reports of memory ability: a web‐based study
Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T., Ling, J., & Parrott, A. (2001) · Human Psychopharmacology Clinical and Experimental · doi:10.1002/hup.345 · Cited by 125
O’Neil, K. M., & Penrod, S. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195369 · Cited by 108
WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures
Reips, U., & Neuhaus, C. (2002) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195449 · Cited by 101
Self-citationHindsight Bias Around the World
POHL, R. F., Bender, M., & Lachmann, G. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026/1618-3169.49.4.270 · Cited by 94
Orne, M. T. (2002) · Prevention & Treatment · doi:10.1037/1522-3736.5.1.535a · Cited by 91
The Web Experimental Psychology Lab: Five years of data collection on the Internet
Reips, U. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195366 · Cited by 89
Self-citationThe Web Experiment Method: Advantages, disadvantages, and solutions
Reips, U. (2000) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-19760 · Cited by 85
Self-citationSurveyWiz and factorWiz: Javascript Web pages that make HTML forms for research on the internet
Birnbaum, M. H. (2000) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207804 · Cited by 63
Hindsight Bias Around the World
POHL, R. F., Bender, M., & Lachmann, G. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.49.4.270 · Cited by 63
Tell me, what did you see? The stimulus on computers
Krantz, J. H. (2000) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207787 · Cited by 45
Rape Myth Acceptance and Judgments of Vulnerability to Sexual Assault: An Internet Experiment
Bohner, G., Danner, U. N., Siebler, F., & Samson, G. B. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026/1618-3169.49.4.257 · Cited by 40
Eichstaedt, J. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195364 · Cited by 39
The Server Side of Psychology Web Experiments
Schmidt, W. C. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50013-7 · Cited by 35
Horswill, M. S., & Coster, M. E. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195346 · Cited by 34
CGI versus JavaScript: A Web experiment on the reversed hindsight bias
Schwarz, S., & Reips, U. (2001) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-19759 · Cited by 33
Self-citationOnline Longitudinal Survey Research: Viability and Participation
Hiskey, S., & Troop, N. A. (2002) · Social Science Computer Review · doi:10.1177/08939302020003003 · Cited by 31
Johnson, H. M. (1923) · Psychological Bulletin · doi:10.1037/h0070676 · Cited by 31
Theory and Techniques of Conducting Web Experiments
Reips, U. (2002) · KOPS (University of Konstanz) · Cited by 23
Self-citationRape Myth Acceptance and Judgments of Vulnerability to Sexual Assault: An Internet Experiment
Bohner, G., Danner, U. N., Siebler, F., & Samson, G. B. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.49.4.257 · Cited by 21
Musch, J., & Klauer, K. C. (2002) · Cited by 18
Orne, M. T. (2002) · Prevention & Treatment · doi:10.1037/1522-3736.5.0035a · Cited by 16
Measuring Differences in Preactivation on the Internet: The Content Category Superiority Effect
Eichstaedt, J. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.49.4.283 · Cited by 15
Research works printed but not linked: 12
- Bamert, T.(2002). Integration von Wahrscheinlichkeiten: Verarbeitung von zwei Wahrscheinlichkeitsinformationen. [Integration of probabilities: Processing two pieces of probability information]. Unpublished master’s thesis, University of Zurich, Switzerland.
- Bosnjak, M.(2001). Participation in non-restricted Web surveys: A typology and explanatory model for item non-response. In U.D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 193– 208 Lengerich, Germany: Pabst Science.
- 10.1037/0735-7028.33.2.148 · doi:10.1037/0735-7028.33.2.148 · Paper page
- Buchanan, T., Reips, U.D.(2001, October 10). Platform-dependent biases in Online Research: Do Mac users really think different? In K. J. Jonas, P. Breuer, B. Schauenburg, & M. Boos (Eds.), Perspectives on Internet Research: Concepts and Methods. Retrieved December 27, 2001, from server3.uni-psych.gwdg.de/gor/contrib/buchanan-tom.
- 10.1177/089443930202000303 · DOI
- 10.1037/0033-295X.107.4.852 · doi:10.1037/0033-295x.107.4.852 · Paper page
- Krantz, J. H.(2001). Stimulus delivery on the Web: What can be presented when calibration isn’t possible. In U.D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 113– 130 Lengerich, Germany: Pabst Science.
- Laugwitz, B.(2001). A Web experiment on color harmony principles applied to computer user interface design. In U.D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 131– 145 Lengerich, Germany: Pabst Science.
- Reips, U.D.(1999). Online research with children. In U.D. Reips, B. Batinic, W. Bandilla, M. Bosnjak, L. Gräf, K. Moser, & A. Werner (Eds.). Current Internet science - trends, techniques, results. Aktuelle Online-Forschung - Trends, Techniken, Ergebnisse. Zürich: Online Press. Retrieved April 7, 2002 from dgof.de/tband99/.
- Reips U.-D., 2002, Social Science Computer Review, 20, 241
- Ruppertsberg, A. I., Givaty, G., Van Veen, H. A. H. C., Bülthoff, H.(2001). Games as research tools for visual perception over the Internet. In U.D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 147– 158 Lengerich, Germany: Pabst.
- Voracek, M., Stieger, S., Gindl, A.(2001). Online replication of Evolutionary Psychology evidence: Sex differences in sexual jealousy in imagined scenarios of mate’s sexual versus emotional infidelity. In U.D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science 91– 112 Lengerich, Germany: Pabst Science.
Cited by
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
“There are many advantages to online experimentation, many of which are detailed in prior work (Reips, 2000, 2002).”
InfluentialEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Metadata
- Authors
- Reips, U.
- Venue
- Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie)
- Publisher
- Hogrefe Publishing Group
- Published
- 2002-10
- Volume
- 49
- Issue
- 4
- Pages
- 243-256
- ISSN
- 0949-3964 (ISSN-L)
- 1618-3169
- 2190-5142
- OpenAlex
- W2140054227
- Cited by
- 642
- References
- 47
- FWCI
- 49.47
- Type
- journal article
- Language
- de
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.