The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index.
Data Collection in the Digital Age: Innovative Alternatives to Student Samples1
Abstract
Online crowdsourcing markets (OCM) are becoming more popular as a source for data collection. In this paper, we examine the consistency of survey results across student samples, consumer panels, and online crowdsourcing markets (specifically Amazon’s Mechanical Turk) both within the United States and outside. We conduct two studies examining the technology acceptance model (TAM) and the expectation– disconfirmation theory (EDT) to explore potential differences in demographics, psychometrics, structural model estimates, and measurement invariances. Our findings indicate that (1) U.S.-based OCM samples provide demographics much more similar to our student and consumer panel samples than the non-U.S.-based OCM samples; (2) both U.S. and non-U.S. OCM samples provide initial psychometric properties (reliability, convergent, and divergent validity) that are similar to those of both student and consumer panels; (3) non-U.S. OCM samples generally provide differences in scale means compared to those of our students, consumer panels, and U.S. OCM samples; and (4) one of the non-U.S. OCM samples refuted the highly replicated and validated TAM model in the relationship of perceived usefulness to behavioral intentions. Although our post hoc analyses isolated some cultural and demographic effects with regard to the non-U.S. samples in Study 1, they did not address the model differences found in Study 2. Specifically, the inclusion of non-U.S. OCM respondents led to statistically significant differences in parameter estimates, and hence to different statistical conclusions. Due to these unexplained differences that exist within the non-U.S. OCM samples, we caution that the inclusion of non-U.S. OCM participants may lead to different conclusions than studies with only U.S. OCM participants. We are unable to conclude whether this is due to of cultural differences, differences in the demographic profiles of non-U.S. OCM participants, or some unexplored factors within the models. Therefore, until further research is conducted to explore these differences in detail, we urge researchers utilizing OCMs with the intention to generalize to U.S. populations focus on U.S.-based participants and exercise caution in using non-U.S. participants. We further recommend that researchers should clearly describe their OCM usage and design (e.g., demographics, participant filters, etc.) procedures. Overall, we find that U.S. OCM samples produced models that lead to similar statistical conclusions as both U.S. students and U.S. consumer panels at a considerably reduced cost.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating structural equation models with unobservable variables and measurement error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.1177/002224378101800104 · Cited by 109,190
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035
(1998) · Cited by 36,049
User Acceptance of Computer Technology: A Comparison of Two Theoretical Models
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989) · Management Science · doi:10.1287/mnsc.35.8.982 · Cited by 25,709
Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance
Cheung, G. W., & Rensvold, R. B. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0902_5 · Cited by 16,280
A Cognitive Model of the Antecedents and Consequences of Satisfaction Decisions
Oliver, R. L. (1980) · Journal of Marketing Research · doi:10.1177/002224378001700405 · Cited by 14,494
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Multivariate Data Analysis, 6th ed
Tatham, R. L., Anderson, R. E., Babin, B. J., Black, W. C., & Hair, J. F. (2006) · Cited by 9,807
How to Write Up and Report PLS Analyses
Chin, W. W. (2010) · Handbook of Partial Least Squares · doi:10.1007/978-3-540-32827-8_29 · Cited by 5,013
Assessing Measurement Invariance in Cross‐National Consumer Research
Steenkamp, J. E. M., & Baumgartner, H. (1998) · Journal of Consumer Research · doi:10.1086/209528 · Cited by 4,750
Byrne, B. M. (2013) · doi:10.4324/9780203805534 · Cited by 4,016
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
A practical guide to factorial validity using PLS-graph: tutorial and annotated example
Gefen, D., & Straub, D. (2005) · Communications of the Association for Information Systems · doi:10.17705/1cais.01605 · Cited by 3,280
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932
A meta-analysis of the technology acceptance model
King, W. R., & He, J. (2006) · Information & Management · doi:10.1016/j.im.2006.05.003 · Cited by 2,906
Nonresponse rates and nonresponse bias in household surveys
Groves, R. M. (2006) · Public Opinion Quarterly · doi:10.1093/poq/nfl033 · Cited by 2,420
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,078
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Leidner, D. E., & Kayworth, T. (2006) · MIS Quarterly · doi:10.2307/25148735 · Cited by 1,554
Survey Errors and Survey Costs.
Faulkenberry, G. D., & Groves, R. M. (1990) · Journal of the American Statistical Association · doi:10.2307/2289634 · Cited by 1,438
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258
Introduction to Factor Analysis: What It Is and How to Do It.
Harper, D., Kim, J.-O., & Mueller, C. W. (1980) · Contemporary Sociology · doi:10.2307/2066279 · Cited by 1,186
Survey Errors and Survey Costs
Groves, R. M. (1989) · Wiley Series in Probability and Statistics · doi:10.1002/0471725277 · Cited by 1,062
Introduction to Factor Analysis
Kim, J.-O., & Mueller, C. (1978) · Sage eBooks · doi:10.4135/9781412984652 · Cited by 1,027
The Effects of Response Rate Changes on the Index of Consumer Sentiment
Curtin, R., Presser, S., & Singer, E. (2000) · Public Opinion Quarterly · doi:10.1086/318638 · Cited by 1,007
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Consequences of Reducing Nonresponse in a National Telephone Survey
Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000) · Public Opinion Quarterly · doi:10.1086/317759 · Cited by 986
Human Research and Data Collection via the Internet
Birnbaum, M. H. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.141601 · Cited by 869
Leverage-Saliency Theory of Survey Participation
Groves, R. M., Singer, E., & Corning, A. (2000) · Public Opinion Quarterly · doi:10.1086/317990 · Cited by 774
World-Wide Web survey research: Benefits, potential problems, and solutions
Schmidt, W. C. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204826 · Cited by 620
More than fun and money. Worker motivation in crowdsourcing - a study on mechanical turk
Kaufmann, N., Schulze, T., & Veit, D. J. (2011) · MADOC (University of Mannheim) · Cited by 576
The "Science of the Sophomore" Revisited: From Conjecture to Empiricism
Gordon, M. E., Slade, L. A., & Schmitt, N. (1986) · The Academy of Management Review · doi:10.2307/258340 · Cited by 574
Toward a Theory-Based Measurement of Culture
Straub, D., Loch, K., Evaristo, R., Karahanna, E., & Srite, M. (2002) · Journal of Global Information Management · doi:10.4018/jgim.2002010102 · Cited by 572
Seriousness checks are useful to improve data validity in online research
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0265-2 · Cited by 524
Sprouse, J. (2011) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 487
Completion Time and Response Order Effects in Web Surveys
Malhotra, N. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn050 · Cited by 332
Mogilner, C., Aaker, J., & Kamvar, S. D. (2012) · Journal of Consumer Research · doi:10.1086/663774 · Cited by 300
Virtual subjects: Using the Internet as an alternative source of subjects and research environment
Smith, M. A., & Leigh, B. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210601 · Cited by 288
How Good Are Humans at Solving CAPTCHAs? A Large Scale Evaluation
Bursztein, E., Bethard, S., Fabry, C., Mitchell, J. C., & Jurafsky, D. (2010) · 2010 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2010.31 · Cited by 254
Self-Affirmation through the Choice of Highly Aesthetic Products
Townsend, C., & Sood, S. (2012) · Journal of Consumer Research · doi:10.1086/663775 · Cited by 238
Opportunity Cost Consideration
Spiller, S. A. (2011) · Journal of Consumer Research · doi:10.1086/660045 · Cited by 236
Using the Amazon Mechanical Turk for transcription of spoken language
Marge, M., Banerjee, S., & Rudnicky, A. I. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494979 · Cited by 224
Internet-Based Data Collection: Promises and Realities
Benfield, J. A., & Szlemko, W. J. (2006) · Journal of research practice · Cited by 213
Ward, M. K., & Broniarczyk, S. M. (2011) · Journal of Consumer Research · doi:10.1086/658166 · Cited by 213
Minimizing Method Bias Through Programmatic Research1
Burton-Jones, A. (2009) · MIS Quarterly · doi:10.2307/20650304 · Cited by 212
Meyerson, P., & Tryon, W. W. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195541 · Cited by 197
Bethlehem, J. G., & Biffignandi, S. (2011) · Wiley eBooks · doi:10.1002/9781118121757 · Cited by 184
Goodman, J. K., & Malkoc, S. A. (2012) · Journal of Consumer Research · doi:10.1086/665047 · Cited by 169
Samples in applied psychology: Over a decade of research in review.
Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011) · Journal of Applied Psychology · doi:10.1037/a0023322 · Cited by 160
Bohannon, J. (2011) · Science · doi:10.1126/science.334.6054.307 · Cited by 140
Effects of Design in Web Surveys
Toepoel, V., Das, M., & van Soest, A. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn060 · Cited by 135
Lewis, I. M., Watson, B. C., & White, K. M. (2008) · Australian Journal of Psychology · doi:10.1080/00049530802105865 · Cited by 134
When Student Samples Make Sense in Logistics Research
Thomas, R. (2011) · Journal of Business Logistics · doi:10.1111/j.2158-1592.2011.01023.x · Cited by 124
Pentz, M. A., & Chou, C. (1994) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.62.3.450 · Cited by 124
Psychological Distance and Subjective Experience: How Distancing Reduces the Feeling of Difficulty
Thomas, M., & Tsai, C. I. (2011) · Journal of Consumer Research · doi:10.1086/663772 · Cited by 124
Illusionary Progress in Loyalty Programs: Magnitudes, Reward Distances, and Step-Size Ambiguity
Bagchi, R., & Li, X. (2010) · Journal of Consumer Research · doi:10.1086/656392 · Cited by 121
Leonhardt, J. M., Keller, L. R., & Pechmann, C. (2011) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2011.01.001 · Cited by 115
Research on the Internet: Validation of a World-Wide Web mediated personality scale
Buchanan, T., & Smith, J. L. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200736 · Cited by 114
Slow down! Insensitivity to rate of consumption leads to avoidable satiation
Galak, J., Kruger, J., & Loewenstein, G. (2013) · Journal of Consumer Research · doi:10.1086/666597 · Cited by 113
Introduction to psychological experiments on the internet
Birnbaum, M. H. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50001-0 · Cited by 107
Profiling the Research Productivity of Tenured Information Systems Faculty at U.S. Institutions1
Dean, D. L., Lowry, P. B., & Humpherys, S. L. (2011) · MIS Quarterly · doi:10.2307/23043486 · Cited by 104
The Presenter's Paradox: Figure 1.
Weaver, K. D., Garcia, S. M., & Schwarz, N. (2012) · Journal of Consumer Research · doi:10.1086/664497 · Cited by 103
The equivalency of computerized and conventional MMPI administration: A critical review
Honaker, L. (1988) · Clinical Psychology Review · doi:10.1016/0272-7358(88)90081-5 · Cited by 81
People helping robots helping people: Crowdsourcing for grasping novel objects
Sorokin, A., Berenson, D., Srinivasa, S. S., & Hebert, M. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5650464 · Cited by 68
The Role of Bolstering and Counterarguing Mind-Sets in Persuasion
Xu, A. J., & Wyer, R. S. (2012) · Journal of Consumer Research · doi:10.1086/661112 · Cited by 61
Safety First? The Role of Emotion in Safety Product Betrayal Aversion
Gershoff, A. D., & Koehler, J. J. (2011) · Journal of Consumer Research · doi:10.1086/658883 · Cited by 58
Decision Quicksand: How Trivial Choices Suck Us In
Sela, A., & Berger, J. A. (2012) · Journal of Consumer Research · doi:10.1086/662997 · Cited by 56
Sun, Y., Wang, N., & Peng, Z. C. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2010.12.007 · Cited by 42
PRIVACY IN ONLINE SOCIAL NETWORKS
Zhang, N., Wang, C., & Xu, Y. (2011) · AIS Electronic Library (AISeL) · Cited by 35
Sun, Y., Wang, N., Yin, C., & Che, T. (2012) · Americas Conference on Information Systems · Cited by 22
The imbibing idiot bias: Consuming alcohol can be hazardous to your (perceived) intelligence
Rick, S. I., & Schweitzer, M. E. (2012) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2012.06.001 · Cited by 20
Grigore, M., & Rosenkranz, C. (2011) · AIS Electronic Library (AISeL) · Cited by 17
Kim, J.-H. (2012) · Americas Conference on Information Systems · Cited by 14
"You Want It When?" How Temporal Dissonance in IT Workers Contributes to Project Failures
Conway, C. M., & Limayem, M. (2011) · AIS Electronic Library (AISeL) · Cited by 14
Self-citationPaolacci, G., Burson, K. A., & Rick, S. I. (2011) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2011.04.005 · Cited by 14
Validating results of human-based electronic services leveraging multiple reviewers
Kern, R., Bauer, C., Thies, H. H., & Satzger, G. (2010) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000022266 · Cited by 13
Crowdsourcing with semantic differentials: a game to investigate the meaning of form
Müller, R., Thoring, K., & Oostinga, R. (2010) · University of Twente Research Information · Cited by 11
DESIGNING RANKING SYSTEMS FOR HOTELS ON TRAVEL SEARCH ENGINES TO ENHANCE USER EXPERIENCE
Ghose, A., Ipeirotis, P. G., & Li, B. (2010) · International Conference on Information Systems · Cited by 6
CROWDSOURCING AS A MOBILE SERVICE - CASE STUDY: PUBLISHING PHOTOGRAPHY
Soliman, W., & Tuunainen, V. K. (2012) · Journal of the Association for Information Systems · Cited by 4
Impact of Online Content on Attitudes and Buying Intentions
Kumi, R., & Limayem, M. (2012) · Cited by 2
Self-citationA new look at user commitment towards information systems: evidence from Amazon's Mechanical Turk
van der Heijden, H. (2012) · Journal of the Association for Information Systems · Cited by 2
Wati, Y., & Davis, F. D. (2012) · Cited by 2
Cited by
Lee, A. R., Son, S.-M., & Kim, K. K. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.011 · Cited by 814
Lowry, P. B., Zhang, J., Wang, C., & Siponen, M. (2016) · Information Systems Research · doi:10.1287/isre.2016.0671 · Cited by 404
Lin, S.-H. (., Ma, J., & Johnson, R. E. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000098 · Cited by 382
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Cross-sectional research: A critical perspective, use cases, and recommendations for IS research
Maier, C., Thatcher, J. B., Grover, V., & Dwivedi, Y. K. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102625 · Cited by 316
Metadata
- Authors
- Steelman, Z. R., Hammer, B. I., & Limayem, M.
- Venue
- MIS Quarterly
- Publisher
- MIS Quarterly
- Published
- 2014-06-01
- Volume
- 38
- Issue
- 2
- Pages
- 355-378
- ISSN
- 0276-7783 (ISSN-L)
- 2162-9730
- OpenAlex
- W1494581773
- Cited by
- 405
- References
- 87
- FWCI
- 29.41
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.