The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Conducting interactive experiments online
Open accessAbstract
Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a potentially valuable complement to laboratory studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Altruistic punishment in humans
Fehr, E., & Gächter, S. (2002) · Nature · doi:10.1038/415137a · Cited by 5,616
Self-citationCooperation and Punishment in Public Goods Experiments
Fehr, E., & Gächter, S. (2000) · American Economic Review · doi:10.1257/aer.90.4.980 · Cited by 4,547
Self-citationEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Subject pool recruitment procedures: organizing experiments with ORSEE
Greiner, B. (2015) · Journal of the Economic Science Association · doi:10.1007/s40881-015-0004-4 · Cited by 2,276
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“in the laboratory and online Pao-lacci et al. 2010; Horton et al. 2011; Goodman et al. 2013).”
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Antisocial Punishment Across Societies
Herrmann, B., Thöni, C., & Gächter, S. (2008) · Science · doi:10.1126/science.1153808 · Cited by 2,038
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 1×
“In addition, to increase the likelihood that participants completed our HIT with care, we only allowed workers with at least 90% of their previous HITs approved by requesters to participate (see Peer et al. 2014 for a detailed discussion of approval rates).”
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,529
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Easy Estimation Methods for Discrete‐Time Duration Models
Jenkins, S. P. (1995) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.1995.tb00031.x · Cited by 892
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
Social heuristics shape intuitive cooperation
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 720
Rand, D. G. (2012) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 683
hroot: Hamburg Registration and Organization Online Tool
Bock, O., Baetge, I., & Nicklisch, A. (2014) · European Economic Review · doi:10.1016/j.euroecorev.2014.07.003 · Cited by 675
Weinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
Reips, U.-D. (2000) · Psychological Experiments on the Internet · doi:10.1016/b978-012099980-4/50005-8 · Cited by 460
Zhou, H., & Fishbach, A. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 459
The economics of altruistic punishment and the maintenance of cooperation
Egas, M., & Riedl, A. (2008) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2007.1558 · Cited by 453
Trust, voluntary cooperation, and socio-economic background: survey and experimental evidence
Gächter, S., Herrmann, B., & Thöni, C. (2004) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2003.11.006 · Cited by 442
Self-citationInequality and visibility of wealth in experimental social networks
Nishi, A., Shirado, H., Rand, D. G., & Christakis, N. A. (2015) · Nature · doi:10.1038/nature15392 · Cited by 370
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 354
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“11 (Stewart et al. 2015) estimate a worker half-life of about seven months and Rand et al.”
CARPENTER, J., & SEKI, E. (2011) · Economic Inquiry · doi:10.1111/j.1465-7295.2009.00268.x · Cited by 283
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251
Young, Selfish and Male: Field Evidence of Social Preferences
List, J. A. (2004) · The Economic Journal · doi:10.1046/j.0013-0133.2003.00180.x · Cited by 217
Cooperation and assortativity with dynamic partner updating
Wang, J., Suri, S., & Watts, D. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1120867109 · Cited by 198 · cited 2×
“3 Thus far interactive research developed its own bespoke software (see: Egas and Riedl 2008; Suri and Watts 2011; Wang et al. 2012; Gallo and Yan 2015; Nishi et al. 2015; Schmelz and Ziegelmeyer 2015; Stagnaro et al. 2016), but recent software is likely to change this trend (see: BreadBoard, MobLab, oTree, SoPHIE, an…”
The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk
Difallah, D. E., Catasta, M., Demartini, G., Ipeirotis, P. G., & Cudré-Mauroux, P. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741685 · Cited by 187
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
Gächter, S., & Herrmann, B. (2011) · European Economic Review · doi:10.1016/j.euroecorev.2010.04.003 · Cited by 177
Self-citationThe effects of reputational and social knowledge on cooperation
Gallo, E., & Chang, Y. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1415883112 · Cited by 146
Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success?
Haigh, M. (2016) · Advances in Cognitive Psychology · doi:10.5709/acp-0193-5 · Cited by 130
From the Lab to the Field: Cooperation among Fishermen
Stoop, J., NOUSSAIR, C. N., & Soest, D. P. V. (2012) · Journal of Political Economy · doi:10.1086/669253 · Cited by 122
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Social Preferences in the Online Laboratory: A Randomized Experiment
Hergueux, J., & Jacquemet, N. (2012) · Experimental Economics · doi:10.1007/s10683-014-9400-5 · Cited by 115
Virtual world experimentation: An exploratory study
Chesney, T., Chuah, S., & Hoffmann, R. F. (2009) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2009.05.026 · Cited by 114
Anderson, J. E., Anderson, J. C., Burks, S. V., Carpenter, J. P., Goette, L. F., Maurer, K., Nosenzo, D., Potter, R., Rocha, K., & Rustichini, A. (2012) · Experimental Economics · doi:10.1007/s10683-012-9327-7 · Cited by 110
Is there selection bias in laboratory experiments? The case of social and risk preferences
Cleave, B. L., Nikiforakis, N., & Slonim, R. L. (2012) · Experimental Economics · doi:10.1007/s10683-012-9342-8 · Cited by 89
Self-Selection into Economics Experiments is Driven by Monetary Rewards
Abeler, J., & Nosenzo, D. (2013) · Experimental Economics · doi:10.1007/s10683-014-9397-9 · Cited by 81
Stagnaro, M. N., Arechar, A. A., & Rand, D. G. (2016) · Cognition · doi:10.1016/j.cognition.2017.01.017 · Cited by 73
Self-citationPfattheicher, S., & Schindler, S. (2015) · European Journal of Personality · doi:10.1002/per.2003 · Cited by 72
Think global, act local: Preserving the global commons
Hauser, O., Hendriks, A., Rand, D. G., & Nowak, M. A. (2016) · Scientific Reports · doi:10.1038/srep36079 · Cited by 64
Belot, M. V. K., Duch, R. M., & Miller, L. (2010) · RePEc: Research Papers in Economics · Cited by 40
What brings your subjects to the lab? A field experiment
Krawczyk, M. W. (2011) · Experimental Economics · doi:10.1007/s10683-011-9277-5 · Cited by 32
Guillén, P., & Veszteg, R. F. (2012) · The Journal of Socio-Economics · doi:10.1016/j.socec.2012.07.002 · Cited by 16
Cited by
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Do Investors Care about Impact?
Heeb, F., Kölbel, J. F., Paetzold, F., & Zeisberger, S. (2023) · The Review of Financial Studies · doi:10.1093/rfs/hhac066 · Cited by 332
Testing the Waters: Behavior across Participant Pools
Snowberg, E., & Yariv, L. (2021) · American Economic Review · doi:10.1257/aer.20181065 · Cited by 273
Moral polarization and out-party hostility in the US political context
Tappin, B. M., & McKay, R. T. (2019) · Journal of Social and Political Psychology · doi:10.5964/jspp.v7i1.1090 · Cited by 79
Metadata
- Authors
- Arechar, A. A., Gächter, S., & Molleman, L.
- Venue
- Experimental Economics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-03
- Volume
- 21
- Issue
- 1
- Pages
- 99-131
- ISSN
- 1386-4157 (ISSN-L)
- 1573-6938
- OpenAlex
- W2587467493
- Cited by
- 340
- References
- 54
- FWCI
- 53.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.