The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
The Online Laboratory: Conducting Experiments in a Real Labor Market
Open accessAbstract
Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials.We argue that online experiments can be just as valid -both internally and externally -as laboratory and field experiments, while requiring far less money and time to design and to conduct.In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results.We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices.We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves.In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats.We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question.We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Axelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497
A Theory of Fairness, Competition, and Cooperation
Fehr, E., & Schmidt, K. M. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399556151 · Cited by 11,264
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Impure Altruism and Donations to Public Goods: A Theory of Warm-Glow Giving
Andreoni, J. (1990) · The Economic Journal · doi:10.2307/2234133 · Cited by 5,825
Cooperation and Punishment in Public Goods Experiments
Fehr, E., & Gächter, S. (2000) · American Economic Review · doi:10.1257/aer.90.4.980 · Cited by 4,547
Behavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
The Handbook of experimental economics
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1645 · Cited by 3,679
Resnick, P., Kuwabara, K., Zeckhauser, R., & Friedman, E. (2000) · Communications of the ACM · doi:10.1145/355112.355122 · Cited by 2,190
Self-citationCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
A Theory of Fairness, Competition and Cooperation
Fehr, E. (1998) · SSRN Electronic Journal · doi:10.2139/ssrn.106228 · Cited by 1,629
CAPTCHA: Using Hard AI Problems for Security
von Ahn, L., Blum, M., Hopper, N. J., & Langford, J. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-39200-9_18 · Cited by 1,447
Mas, A., & Moretti, E. (2009) · American Economic Review · doi:10.1257/aer.99.1.112 · Cited by 1,280
Behavioral game theory, experiments in strategic interaction
Canté, F. (2004) · Cuadernos de Economía · Cited by 1,265
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1985) · Behavioral Decision Making · doi:10.1007/978-1-4613-2391-4_2 · Cited by 1,252
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165
God Is Watching You: Priming God Concepts Increases Prosocial Behavior in an Anonymous Economic Game
Shariff, A. F., & Norenzayan, A. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01983.x · Cited by 1,136
The value of reputation on eBay: A controlled experiment
Resnick, P., Zeckhauser, R., Swanson, J., & Lockwood, K. (2006) · Experimental Economics · doi:10.1007/s10683-006-4309-2 · Cited by 1,118
Self-citationLab Experiments Are a Major Source of Knowledge in the Social Sciences
Falk, A., & Heckman, J. J. (2009) · Science · doi:10.1126/science.1168244 · Cited by 1,033
The Scientific Research Potential of Virtual Worlds
Bainbridge, W. S. (2007) · Science · doi:10.1126/science.1146930 · Cited by 935
Gender Differences in Competition: Evidence\\ From a Matrilineal and a Patriarchal Society
(2009) · Econometrica · doi:10.3982/ecta6690 · Cited by 926
Social Identity and Preferences
Benjamin, D. J., Choi, J. J., & Strickland, A. J. (2010) · American Economic Review · doi:10.1257/aer.100.4.1913 · Cited by 681
Utility data annotation with Amazon Mechanical Turk
Sorokin, A., & Forsyth, D. (2008) · 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562953 · Cited by 616
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601
Self-citationField experiments in economics: The past, the present, and the future
Levitt, S. D., & List, J. A. (2009) · European Economic Review · doi:10.1016/j.euroecorev.2008.12.001 · Cited by 590
Auctions on the Internet: What’s Being Auctioned, and How?
Lucking‐Reiley, D. (2000) · The Journal of Industrial Economics · doi:10.1111/1467-6451.00122 · Cited by 561
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494
Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States
Bohnet, I., Greig, F., Herrmann, B., & Zeckhauser, R. (2008) · American Economic Review · doi:10.1257/aer.98.1.294 · Cited by 481
Self-citationSocial Connections and Incentives in the Workplace: Evidence From Personnel Data
(2009) · Econometrica · doi:10.3982/ecta6496 · Cited by 407
Stanley, J. C. (1965) · The School Review · doi:10.1086/442740 · Cited by 406
Breaking monotony with meaning: Motivation in crowdsourcing markets
Chandler, D., & Kapelner, A. (2013) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2013.03.003 · Cited by 398
The Decision Maker Matters: Individual Versus Group Behaviour in Experimental Beauty‐Contest Games
Kocher, M. G., & Sutter, M. (2005) · The Economic Journal · doi:10.1111/j.1468-0297.2004.00966.x · Cited by 298
Hot vs. cold: Sequential responses and preference stability in experimental games
Brandts, J., & Charness, G. (2000) · Experimental Economics · doi:10.1007/bf01669197 · Cited by 255
Gender Differences in Competition: Evidence from a Matrilineal and a Patriarchal Society
Gneezy, U., Leonard, K., & List, J. (2008) · National Bureau of Economic Research · doi:10.3386/w13727 · Cited by 248
TurKit: tools for iterative tasks on mechanical Turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600159 · Cited by 237
What Determines Giving to Hurricane Katrina Victims? Experimental Evidence on Racial Group Loyalty
Fong, C. M., & Luttmer, E. F. P. (2009) · American Economic Journal: Applied Economics · doi:10.1257/app.1.2.64 · Cited by 195
Measuring conditional cooperation: a replication study in Russia
Herrmann, B., & Thöni, C. (2009) · Experimental Economics · doi:10.1007/s10683-008-9197-1 · Cited by 166
Horton, J. J. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-17572-5_45 · Cited by 158
Self-citationField Experiments in Economics: The Past, The Present, and The Future
Levitt, S. D., & List, J. A. (2008) · National Bureau of Economic Research · doi:10.3386/w14356 · Cited by 133
Gender differences in competition: Evidence from a matrilineal and a patriarchal societyPreprint
Gneezy, U., Leonard, K. L., & List, J. A. (2008) · RePEc: Research Papers in Economics · Cited by 121
Religious Identity and Economic Behavior
Benjamin, D. J., Choi, J. J., & Fisher, G. (2010) · National Bureau of Economic Research · doi:10.3386/w15925 · Cited by 113
The condition of the Turking class: Are online employers fair and honest?
Horton, J. J. (2010) · Economics Letters · doi:10.1016/j.econlet.2010.12.007 · Cited by 92
Self-citationInternet cautions: Experimental games with internet partners
Eckel, C. C., & Wilson, R. K. (2006) · Experimental Economics · doi:10.1007/s10683-006-4307-4 · Cited by 43
Chilton, L. B., Sims, C. T., Goldman, M., Little, G., & Miller, R. C. (2009) · doi:10.1145/1600150.1600162 · Cited by 30
The Wages of Pay Cuts: Evidence from a Field ExperimentPreprint
Chen, D. L., & Horton, J. J. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1443526 · Cited by 10
Self-citation
Cited by
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
oTree—An open-source platform for laboratory, online, and field experiments
Chen, D. L., Schonger, M., & Wickens, C. (2016) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2015.12.001 · Cited by 1,889
Social contagion theory: examining dynamic social networks and human behavior
Christakis, N. A., & Fowler, J. H. (2013) · Statistics in Medicine · doi:10.1002/sim.5408 · Cited by 1,303
Prior exposure increases perceived accuracy of fake news
Pennycook, G., Cannon, T. D., & Rand, D. G. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000465 · Cited by 1,268
Metadata
- Authors
- Horton, J., Rand, D., & Zeckhauser, R.
- Venue
- National Bureau of Economic Research
- Publisher
- National Bureau of Economic Research
- Published
- 2010-05
- OpenAlex
- W2164976104
- Cited by
- 388
- References
- 47
- Type
- report
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.