The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Gorilla in our midst: An online behavioral experiment builder
Open accessAbstract
Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing-an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test-retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform's aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant's own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the "conflict network" effect in all these populations, demonstrating the platform's capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Cook, R. J., Dickens, B. M., & Fathalla, M. F. (2003) · Reproductive Health and Human Rights · doi:10.1093/acprof:oso/9780199241323.003.0025 · Cited by 10,769
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
Testing the efficiency and independence of attentional networks
Fan, J., McCandliss, B. D., Sommer, T., Raz, A., & Posner, M. I. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902317361886 · Cited by 3,717
Handbook of self-regulation: research, theory, and applications
(2005) · Choice Reviews Online · doi:10.5860/choice.42-3719 · Cited by 3,645
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Spectre Attacks: Exploiting Speculative Execution
Kocher, P., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., Schwarz, M., & Yarom, Y. (2018) · arXiv · doi:10.48550/arxiv.1801.01203 · Cited by 1,850
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,778
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments
Stoet, G. (2017) · Teaching of Psychology · doi:10.1177/0098628316677643 · Cited by 1,299
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Effective Analysis of Reaction Time Data
Whelan, R. (2008) · The Psychological Record · doi:10.1007/bf03395630 · Cited by 835
Turning software into a service
Turner, M., Budgen, D., & Brereton, P. (2003) · Computer · doi:10.1109/mc.2003.1236470 · Cited by 529
Lange, K., Kühn, S., & Filevich, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130834 · Cited by 474
Computation of measures of effect size for neuroscience data sets
Hentschke, H., & Stüttgen, M. C. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07902.x · Cited by 410
Jansen, J., Bogaerts, B., Devriendt, J., Janssens, G., & Denecker, M. (2016) · International Joint Conference on Artificial Intelligence · Cited by 303
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301
Appraising the ANT: Psychometric and theoretical considerations of the Attention Network Test.
MacLeod, J. W., Lawrence, M. A., McConnell, M. M., Eskes, G. A., Klein, R. M., & Shore, D. I. (2010) · Neuropsychology · doi:10.1037/a0019803 · Cited by 296
Attentional control and self-regulation.
Rueda, M. R., Posner, M. I., & Rothbart, M. K. (2011) · Dialnet (Universidad de la Rioja) · Cited by 259
de Leeuw, J. R., & Motz, B. A. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0567-2 · Cited by 252
Building Experiments with PsychoPy
Wang, Z. (2021) · Eye-Tracking with Python and Pylink · doi:10.1007/978-3-030-82635-2_2 · Cited by 208
Online webcam-based eye tracking in cognitive science: A first look
Semmelmann, K., & Weigelt, S. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0913-7 · Cited by 196
Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments
Reimers, S., & Stewart, N. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0471-1 · Cited by 194
QRTEngine: An easy solution for running online reaction time experiments using Qualtrics
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0530-7 · Cited by 166
Big data in psychology: A framework for research advancement.
Adjerid, I., & Kelley, K. (2018) · American Psychologist · doi:10.1037/amp0000190 · Cited by 140
Fantastic Timers and Where to Find Them: High-Resolution Microarchitectural Attacks in JavaScript
Schwarz, M., Maurice, C., Gruss, D., & Mangard, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70972-7_13 · Cited by 133
Neural processing around 200 ms after stimulus-onset correlates with subjective visual awareness
Koivisto, M., & Grassini, S. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.02.024 · Cited by 120
Pollock, L. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0938-y · Cited by 90
Building experiments in PsychoPy
Peirce, J. W., Hirst, R. J., & MacAskill, M. (2022) · Cited by 88
General Markers of Conscious Visual Perception and Their Timing
Rutiku, R., Aru, J., Tallinn, A., & Bachmann, T. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00023 · Cited by 75
Managing crowdsourced human computation: a tutorial
Ipeirotis, P. G., & Paritosh, P. K. (2011) · International World Wide Web Conference (WWW) · doi:10.1145/1963192.1963314 · Cited by 63
Lumsden, J., Skinner, A., Coyle, D., Lawrence, N., & Munafo, M. R. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8473 · Cited by 61
Presentation accuracy of Web animation methods
Schmidt, W. C. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195365 · Cited by 59
Zloteanu, M., Harvey, N., Tuckett, D., & Livan, G. (2018) · PLOS ONE · doi:10.1371/journal.pone.0209071 · Cited by 56
Listeners form average-based representations of individual voice identities
Lavan, N., Knight, S., & McGettigan, C. (2019) · Nature Communications · doi:10.1038/s41467-019-10295-w · Cited by 47
Garaizar, P., & Reips, U. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1126-4 · Cited by 43
Miller, R. E., Schmidt, K., Kirschbaum, C., & Enge, S. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1036-5 · Cited by 42
Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments
Reimers, S. J., & Stewart, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0758-5 · Cited by 41
Measuring narrative engagement: The heart tells the story
Richardson, D. C., Griffin, N. K., Zaki, L., Stephenson, A., Yan, J., Curry, T., Noble, R., Hogan, J., Skipper, J. I., & Devlin, J. T. (2018) · doi:10.1101/351148 · Cited by 39
The Server Side of Psychology Web Experiments
Schmidt, W. C. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50013-7 · Cited by 35
Methods to Test Visual Attention Online
Yung, A., Cardoso-Leite, P., Dale, G., Bavelier, D., & Green, C. S. (2015) · Journal of Visualized Experiments · doi:10.3791/52470 · Cited by 26
Presentation Accuracy of the Web Revisited: Animation Methods in the HTML5 Era
Garaizar, P., Vadillo, M. A., & López–de–Ipiña, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109812 · Cited by 25
Quantifying the web browser ecosystem
Ferdman, S., Minkov, E., Bekkerman, R., & Gefen, D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179281 · Cited by 11
Design Strategies for Efficient Access to Mobile Device Users via Amazon Mechanical Turk
Jacques, J. T., & Kristensson, P. O. (2017) · doi:10.1145/3139243.3139247 · Cited by 8
Usher‐Smith, J. A., Masson, G., Mills, K., Sharp, S. J., Sutton, S. R., Klein, W. M. P., & Griffin, S. J. (2018) · BMC Public Health · doi:10.1186/s12889-018-5712-2 · Cited by 6
Studies of cross-lingual long-term primingPreprint
Poort, E. D., & Rodd, J. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ert8k · Cited by 4
Jasmin, K. M., Dick, F., Holt, L. L., & Tierney, A. T. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/263079 · Cited by 3
Cited by
Research Methods in Applied Linguistics
(2021) · Research methods in applied linguistics · doi:10.1075/rmal · Cited by 2,793
The timing mega-study: comparing a range of experiment generators, both lab-based and online
Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020) · PeerJ · doi:10.7717/peerj.9414 · Cited by 546
Realistic precision and accuracy of online experiment platforms, web browsers, and devices
Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01501-5 · Cited by 341
Self-citationSurvey-software implicit association tests: A methodological and empirical analysis
Carpenter, T. P., Pogacar, R., Pullig, C., Kouril, M., Aguilar, S., LaBouff, J., Isenberg, N., & Chakroff, A. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01293-3 · Cited by 310
jsPsych: Enabling an Open-Source Collaborative Ecosystem of Behavioral Experiments
de Leeuw, J. R., Gilbert, R. A., & Luchterhandt, B. (2023) · Journal of Open Source Software · doi:10.21105/joss.05351 · Cited by 285
Metadata
- Authors
- Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K.
- Venue
- Behavior Research Methods
- Published
- 2019-04-23
- Volume
- 52
- Issue
- 1
- Pages
- 388-407
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2897312833
- Cited by
- 1,862
- References
- 50
- FWCI
- 148.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.