An examination of the equivalency of self-report measures obtained from crowdsourced versus undergraduate student samples
Open accessAbstract
Increasingly, researchers have begun to explore the potential of the Internet to reach beyond the traditional undergraduate sample. In the present study, we sought to compare the data obtained from a conventional undergraduate college-student sample to data collected via two online survey recruitment platforms. In order to examine whether the data sampled from the three populations were equivalent, we conducted a test of equivalency using inferential confidence intervals-an approach that differs from the more traditional null hypothesis significance testing. The results showed that the data obtained via the two online recruitment platforms, the Amazon Mechanical Turk crowdsourcing site and the virtual environment of Second Life, were statistically equivalent to the data obtained from the college sample, on the basis of means of standardized measures of psychological stress and sleep quality. Additionally, correlations between the sleep and stress measures were not statistically different between the groups. These results, along with practical considerations for the use of these recruitment platforms, are discussed, and recommendations for other researchers who may be considering the use of these platforms are provided.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The majority of studies (n=15) were conducted in North America.
“The geographical location of the studies varied widely with the majority of (n=15) studies conducted in North America [37,44,46-58].”
From A systematic review and meta-analysis of poor sleep, insomnia symptoms and stress in undergraduate students (2022) · cites this paper as [49]
Cites
The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research
Buysse, D. J., Reynolds, C. F., Monk, T. H., Berman, S. R., & Kupfer, D. J. (1989) · Psychiatry Research · doi:10.1016/0165-1781(89)90047-4 · Cited by 35,234 · see citing works
A Global Measure of Perceived Stress
Cohen, S., Kamarck, T., & Mermelstein, R. (1983) · Journal of Health and Social Behavior · doi:10.2307/2136404 · Cited by 32,807 · see citing works
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965 · see citing works
What's wrong with Bonferroni adjustments
Perneger, T. V. (1998) · BMJ · doi:10.1136/bmj.316.7139.1236 · Cited by 5,900 · see citing works
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,169 · see citing works
(1988) · Cited by 4,111 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Practical Assessment, Research & Evaluation, 2000-2001.
Rudner, L. M., & Schafer, W. D. (2001) · Practical assessment, research & evaluation · Cited by 3,899 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · see citing works
Schuirmann, D. J. (1987) · Journal of Pharmacokinetics and Biopharmaceutics · doi:10.1007/bf01068419 · Cited by 2,696 · see citing works
Wright, K. B. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00259.x · Cited by 2,550 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
A farewell to Bonferroni: the problems of low statistical power and publication bias
Nakagawa, S. (2004) · Behavioral Ecology · doi:10.1093/beheco/arh107 · Cited by 2,102 · see citing works
The social psychology of health
(1989) · Choice Reviews Online · doi:10.5860/choice.26-5686 · Cited by 1,797 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
Do multiple outcome measures require p-value adjustment?
Feise, R. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-8 · Cited by 1,299 · see citing works
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · see citing works
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · see citing works
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004 · see citing works
Kraut, R. E., Olson, J., Banaji, M. R., Bruckman, A. S., Cohen, J., & Couper, M. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.105 · Cited by 922 · see citing works
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798 · see citing works
Understanding Equivalence and Noninferiority Testing
Walker, E., & Nowacki, A. S. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1513-8 · Cited by 751 · see citing works
Yee, N., Bailenson, J. N., Urbanek, M., Chang, F., & Merget, D. (2007) · CyberPsychology & Behavior · doi:10.1089/cpb.2006.9984 · Cited by 681 · see citing works
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · see citing works
Using the Internet for psychological research: Personality testing on the World Wide Web
Buchanan, T., & Smith, J. L. (1999) · British Journal of Psychology · doi:10.1348/000712699161189 · Cited by 596 · see citing works
Using significance tests to evaluate equivalence between two experimental groups.
Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.113.3.553 · Cited by 568 · see citing works
Web Survey Methods: Introduction
Couper, M. P., & Miller, P. G. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn066 · Cited by 567 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · doi:10.1145/1600150.1600175 · Cited by 544 · see citing works
Sprouse, J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 492 · see citing works
Toward a Definition of “Virtual Worlds”
Bell, M. W. (1970) · Journal of Virtual Worlds Research · doi:10.4101/jvwr.v1i1.283 · Cited by 459 · see citing works
Tryon, W. (2001) · Psychological methods · doi:10.1037/1082-989x.6.4.371 · Cited by 423 · see citing works
Web-Based Recruiting for Health Research Using a Social Networking Site: An Exploratory Study
Fenner, Y., Garland, S. M., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., Gunasekaran, B., & Wark, J. D. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1978 · Cited by 406 · see citing works
Weigold, A., Weigold, I. K., & Russell, E. J. (2013) · Psychological Methods · doi:10.1037/a0031607 · Cited by 333 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1× · see citing works
“In their recent analysis, Stewart et al. (2015) estimated that the number of MTurk workers who participate in research at any given time is only 7,300 — considerably smaller than the total number of MTurk workers advertised by Amazon.”
BackgroundThe Use of the Internet in Psychological Research: Comparison of Online and Offline Questionnaires
Riva, G., Teruzzi, T., & Anolli, L. (2003) · CyberPsychology & Behavior · doi:10.1089/109493103321167983 · Cited by 304 · see citing works
Using the Amazon Mechanical Turk for transcription of spoken language
Marge, M., Banerjee, S., & Rudnicky, A. I. (2010) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2010.5494979 · Cited by 272 · see citing works
A mazon Mechanical Turk and the commodification of labour
Bergvall‐Kåreborn, B., & Howcroft, D. (2014) · New Technology Work and Employment · doi:10.1111/ntwe.12038 · Cited by 258 · see citing works
psiTurk: An open-source framework for conducting replicable behavioral experiments online
Gureckis, T. M., Martin, J. B., McDonnell, J. V., Rich, A. S., Markant, D. B., Coenen, A., Halpern, D., Hamrick, J. B., & Chan, P. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0642-8 · Cited by 256 · see citing works
Simpson's paradox and calculation of number needed to treat from meta-analysis
Cates, C. J. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-1 · Cited by 251 · see citing works
Comparing online and lab methods in a problem-solving experiment
Dandurand, F., Shultz, T. R., & Onishi, K. H. (2008) · Behavior Research Methods · doi:10.3758/brm.40.2.428 · Cited by 215 · see citing works
Simons, M. J. P., Cohen, A. A., & Verhulst, S. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043088 · Cited by 200 · see citing works
Meyerson, P., & Tryon, W. W. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195541 · Cited by 197 · see citing works
Simons, D. J., & Chabris, C. F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051876 · Cited by 180 · see citing works
A Framework for the Development of Computerized Adaptive Tests
Thompson, N. A., & Weiss, D. A. (2020) · doi:10.7275/wqzt-9427 · Cited by 171 · see citing works
Wintre, M. G., North, C., & Sugar, L. A. (2001) · Canadian Psychology/Psychologie canadienne · doi:10.1037/h0086893 · Cited by 164 · see citing works
USE OF SECOND LIFE IN K-12 AND HIGHER EDUCATION: A Review of Research
Inman, C., Wright, V. H., & Hartman, J. A. (2011) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 158 · see citing works
Morgan, A. J., Jorm, A. F., & Mackinnon, A. J. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2262 · Cited by 150 · see citing works
Insuring sample equivalence across internet and paper-and-pencil assessments
Epstein, J., Klinkenberg, W. D., Wiley, D. C., & McKinley, L. (2001) · Computers in Human Behavior · doi:10.1016/s0747-5632(01)00002-4 · Cited by 150 · see citing works
Beuckelaer, A. D., & Lievens, F. (2008) · Applied Psychology · doi:10.1111/j.1464-0597.2008.00350.x · Cited by 150 · see citing works
The Fallacy of Online Surveys: No Data Are Better Than Bad Data
Duda, M. D., & Nobile, J. L. (2010) · Human Dimensions of Wildlife · doi:10.1080/10871200903244250 · Cited by 133 · see citing works
Wired but not WEIRD: The promise of the Internet in reaching more diverse samples
Gosling, S. D., Sandy, C. J., John, O. P., & Potter, J. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000300 · Cited by 119 · see citing works
Lewis, I. M., Watson, B. C., & White, K. M. (2008) · Australian Journal of Psychology · doi:10.1080/00049530802105865 · Cited by 114 · see citing works
Avatar-driven self-disclosure: The virtual me is the actual me
Hooi, R., & Cho, H. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.06.019 · Cited by 101 · see citing works
Buchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., & Scholey, A. B. (2005) · Behavior Research Methods · doi:10.3758/bf03206409 · Cited by 98 · see citing works
Collecting response times using Amazon Mechanical Turk and Adobe Flash
Simcox, T., & Fiez, J. A. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0345-y · Cited by 97 · see citing works
Social engagement and user immersion in a socially based virtual world
Grinberg, A. M., Careaga, J. S., Mehl, M. R., & O'Connor, M. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.04.008 · Cited by 80 · see citing works
Tryon, W. W., & Lewis, C. (2008) · Psychological Methods · doi:10.1037/a0013158 · Cited by 79 · see citing works
Using Facebook as a Subject Recruitment Tool for Survey-Experimental ResearchPreprint
Samuels, D., & Zucco, C. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2101458 · Cited by 78 · see citing works
A validation of well-being and happiness surveys for administration via the Internet
Howell, R. T., Rodzon, K. S., Kurai, M., & Sanchez, A. H. (2010) · Behavior Research Methods · doi:10.3758/brm.42.3.775 · Cited by 78 · see citing works
Online- versus paper-pencil-version of a high potential intelligence test
Preckel, F., & Thiemann, H. (2003) · Swiss Journal of Psychology · doi:10.1024//1421-0185.62.2.131 · Cited by 74 · see citing works
Recommendations for applying tests of equivalence
Cribbie, R. A., Gruman, J. A., & Arpin‐Cribbie, C. A. (2003) · Journal of Clinical Psychology · doi:10.1002/jclp.10217 · Cited by 71 · see citing works
Tests for Treatment Group Equality When Data are Nonnormal and Heteroscedastic
Cribbie, R. A., Wilcox, R. R., Bewell, C., & Keselman, H. J. (2007) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1177992660 · Cited by 54 · see citing works
Lee, J.-E. R. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.03.040 · Cited by 49 · see citing works
Steele, R. M., Mummery, W. K., & Dwyer, T. (2009) · Health Education & Behavior · doi:10.1177/1090198109335802 · Cited by 48 · see citing works
Greiner, B., Caravella, M., & Roth, A. E. (2014) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.01.011 · Cited by 48 · see citing works
Rusticus, S. A., & Lovato, C. Y. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/d5wf-5p77 · Cited by 46 · see citing works
Deception in avatar-mediated virtual environment
Hooi, R., & Cho, H. (2012) · Computers in Human Behavior · doi:10.1016/j.chb.2012.09.004 · Cited by 34 · see citing works
McLeod, P. L., Liu, Y., & Axline, J. E. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.06.025 · Cited by 33 · see citing works
Using Second Life to Teach About Marketing in Second Life
Halvorson, W., Ewing, M. T., & Windisch, L. (2011) · Journal of Marketing Education · doi:10.1177/0273475311410854 · Cited by 31 · see citing works
Tawa, J., Negrón, R., Suyemoto, K. L., & Carter, A. S. (2015) · Group Processes & Intergroup Relations · doi:10.1177/1368430214561694 · Cited by 29 · see citing works
Bell, M. W., Castronova, E., & Wagner, G. G. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1418562 · Cited by 13 · see citing works
Keelan, J. E., Beard, L., Morra, D. J., Busch, V., Atkinson, K. M., & Wilson, K. R. (2015) · Health Policy and Technology · doi:10.1016/j.hlpt.2015.04.004 · Cited by 11 · see citing works
Bell, M. W., Castronova, E., & Wagner, G. G. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1188562 · Cited by 11 · see citing works
The Effectiveness of Survey Recruitment Methods in Second Life
Dean, E., Cook, S., Murphy, J., & Keating, M. (2011) · Social Science Computer Review · doi:10.1177/0894439311410024 · Cited by 10 · see citing works
Swicegood, J., & Haque, S. N. (2015) · Journal of Virtual Worlds Research · doi:10.4101/jvwr.v8i1.7097 · Cited by 4 · see citing works
Reconsidering Site and Self: Methodological Frameworks for Virtual-World Research
Martey, R. M., & Shiflett, K. (2012) · International journal of communication · Cited by 4 · see citing works
Virtual Worlds: In-world Survey Methodological Considerations
Anstadt, S. P., Bradley, S., & Burnette, A. (2013) · Journal of Technology in Human Services · doi:10.1080/15228835.2013.784107 · Cited by 4 · see citing works
REPRESENTASI PRIBUMI DALAM IKLAN SURAT KABAR PANDJI POESTAKA 1940-1941
Urfan, N. F. (2012) · Jurnal Komunikasi · doi:10.20885/komunikasi.vol6.iss2.art3 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 89 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Mortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441 · see citing works
Gardani, M., Bradford, D. R. R., Russell, K., Allan, S., Beattie, L., Ellis, J. G., & Akram, U. (2022) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101565 · Cited by 368 · see citing works
Jung, Y., & Takeuchi, N. (2017) · Human Relations · doi:10.1177/0018726717715075 · Cited by 146 · see citing works
Dupuis, M., Meier, E., & Cuneo, F. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1103-y · Cited by 123 · see citing works
Ibarra, J., Agas, J. M., Lee, M., Pan, J. L., & Buttenheim, A. M. (2018) · JMIR Research Protocols · doi:10.2196/resprot.8071 · Cited by 87 · see citing works
Kraemer, J. D., Strasser, A. A., Lindblom, E. N., NIAURA, R. S., & Mays, D. M. (2017) · Preventive Medicine · doi:10.1016/j.ypmed.2017.07.006 · Cited by 66 · see citing works
Newson, M., Buhrmester, M. D., Xygalatas, D., & Whitehouse, H. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.38413 · Cited by 48 · see citing works
Weigold, A., & Weigold, I. K. (2021) · Social Science Computer Review · doi:10.1177/08944393211006847 · Cited by 49 · see citing works
Probst, T., Pryss, R. C., Langguth, B., Spiliopoulou, M., Landgrebe, M., Vesala, M., Harrison, S., Schobel, J., Reichert, M., Stach, M., et al. (2017) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2017.00113 · Cited by 39 · see citing works
Fowler, J. A., Gullo, M. J., & Elphinston, R. A. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.109965 · Cited by 38 · see citing works
Is subclinical gambling really subclinical?
Weinstock, J., April, L. M., & Kallmi, S. (2017) · Addictive Behaviors · doi:10.1016/j.addbeh.2017.05.014 · Cited by 33 · see citing works
Stress and sleep remain significant predictors of health after controlling for negative affect
Benham, G., & Charak, R. (2019) · Stress and Health · doi:10.1002/smi.2840 · Cited by 31 · see citing works
Self-citationBenham, G. (2019) · Sleep Health · doi:10.1016/j.sleh.2019.07.007 · Cited by 27 · see citing works
Self-citationGehrt, T. B., Nielsen, N. P., Hoyle, R. H., Rubin, D. C., & Berntsen, D. (2022) · Journal of Applied Research in Memory and Cognition · doi:10.1037/h0101869 · Cited by 25 · see citing works
Philip, J. (2021) · Journal of Management & Organization · doi:10.1017/jmo.2021.1 · Cited by 20 · see citing works
Seo, H., Xiong, A., Lee, S., & Lee, D. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19344 · Cited by 21 · see citing works
Badr, H., Oluyomi, A., Fahmideh, M. A., Raza, S., Zhang, X., El-Mubasher, O., & Amos, C. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-044642 · Cited by 17 · see citing works
Benham, G. (2021) · Stress and Health · doi:10.1002/smi.3055 · Cited by 19 · see citing works
Self-citationEsmaeilzadeh, P., Mirzaei, T., & Maddah, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104058 · Cited by 16 · see citing works
Superwoman, Racial Identity, and Teen Dating Violence Victimization among Young Black Women
Debnam, K. J., Milam, A. J., & Finigan‐Carr, N. (2021) · Journal of Interpersonal Violence · doi:10.1177/08862605211021984 · Cited by 14 · see citing works
Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasks
Krendl, A. C., Hugenberg, K., & Kennedy, D. P. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02152-y · Cited by 14 · see citing works
Because you had a bad day: the role of negative affect and justification in self-control failure
Heiland, A. M., & Veilleux, J. C. (2022) · Cognition & Emotion · doi:10.1080/02699931.2022.2067134 · Cited by 11 · see citing works
Tejwani, R., Wang, H. S., Lloyd, J. C., Kokorowski, P. K., Nelson, C. P., & Routh, J. C. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.09.119 · Cited by 10 · see citing works
Zhang, S., Schaub, F., Feng, Y., & Sadeh, N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15742-5_71 · Cited by 9 · see citing works
Barron, D. S., Voracek, M., Tran, U. S., Ong, H. S., Morgan, K. D., Towell, T., & Swami, V. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.08.070 · Cited by 8 · see citing works
Dorina, I., Nikpour, A., Mullan, B. A., & Uren, H. V. (2025) · Foods · doi:10.3390/foods14111970 · Cited by 7 · see citing works
Validation of the Facebook Addiction Questionnaire
Elphinston, R. A., Gullo, M. J., & Connor, J. P. (2022) · Personality and Individual Differences · doi:10.1016/j.paid.2022.111619 · Cited by 7 · see citing works
Fissel, E. R., Fisher, B. S., & Wilcox, P. (2023) · Criminal Justice and Behavior · doi:10.1177/00938548231180636 · Cited by 7 · see citing works
Effects of mental context reinstatement on accuracy and recollective experience
Faber, P., Nielsen, N. P., & Berntsen, D. (2023) · Applied Cognitive Psychology · doi:10.1002/acp.4097 · Cited by 7 · see citing works
Can Carelessness Be Captured? Assessing Careless Responding in Attitudes Toward Novel Stimuli
Conway, M. A., O'Shea, B., Redford, L., Pogge, G. C., Klein, R. A., & Ratliff, K. A. (2019) · Social Cognition · doi:10.1521/soco.2019.37.5.468 · Cited by 6 · see citing works
A 10 Nation Exploration of Trustworthiness in The Workplace
Kwantes, C. T., & Kartolo, A. B. (2021) · Interpersona An International Journal on Personal Relationships · doi:10.5964/ijpr.5639 · Cited by 5 · see citing works
MacLellan, A., Fairchild, G., & Button, K. (2025) · Cognitive Therapy and Research · doi:10.1007/s10608-025-10636-8 · Cited by 4 · see citing works
Effect sizes for equivalence testing: Incorporating the equivalence interval
Gutierrez, N. M., & Cribbie, R. A. (2023) · Methods in Psychology · doi:10.1016/j.metip.2023.100127 · Cited by 5 · see citing works
Walker, R. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00232-2 · Cited by 5 · see citing works
Completing Surveys With Different Item Formats: Testing Equivalence
Weigold, A., Weigold, I. K., Dykema, S. A., & Drakeford, N. M. (2020) · Social Science Computer Review · doi:10.1177/0894439320955143 · Cited by 4 · see citing works
Genetics, imaging, and cognition
Whelan, R., Cao, Z., O’Halloran, L., & Pennie, B. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-815298-0.00027-7 · Cited by 4 · see citing works
Hill, M. A., Heiland, A. M., & Veilleux, J. C. (2021) · Current Psychology · doi:10.1007/s12144-021-01380-9 · Cited by 4 · see citing works
The Impact of Health Literacy on Trust in Online Plastic Surgery Content: A Nationwide Analysis
Kim, E., Foppiani, J., Alvarez, A. H., Valentine, L., Weidman, A. A., Bustos, V. P., Lin, S. J., & Lee, B. T. (2024) · Journal of Surgical Research · doi:10.1016/j.jss.2024.03.027 · Cited by 3 · see citing works
Crowdsourcing utility estimation for spina bifida in the general population
Jiang, R., Inouye, B. M., Wang, H. S., Tejwani, R., & Routh, J. C. (2017) · Journal of Pediatric Rehabilitation Medicine · doi:10.3233/prm-170453 · Cited by 3 · see citing works
Gehrt, T. B., Nielsen, N. P., Hoyle, R. H., Rubin, D. C., & Berntsen, D. (2021) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2021.07.004 · Cited by 3 · see citing works
Security Mechanisms on Web-Based Exams in Introductory Statistics Community College Courses
Feinman, Y. (2018) · Journal of Social Behavioral and Health Sciences · doi:10.5590/jsbhs.2018.12.1.11 · Cited by 3 · see citing works
Geospatial Analytics of Pedestrian Safety in Tanah Abang
Yusuf, D. S., Pradana, K. A., Aditya, T., Jong-Gun, L., Amin, I., & Muztahid, M. R. (2018) · International Conference on Science and Technology (ICST) · doi:10.1109/icstc.2018.8528599 · Cited by 2 · see citing works
Campoli, J., & Cummings, J. A. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.11.002 · Cited by 2 · see citing works
Community perspectives on difference of sex development (DSD) diagnoses: A crowdsourced survey
Alkazemi, M. H., Johnston, A. W., Meglin, D. E., Adkins, D. W., & Routh, J. C. (2020) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2020.03.023 · Cited by 2 · see citing works
Examining the Utility of Behavioral Economic Demand in Addiction Science
Strickland, J. C. (2019) · UKnowledge (University of Kentucky) · doi:10.13023/etd.2019.030 · Cited by 1 · see citing works
Effect Sizes for Equivalence Testing: Incorporating the Equivalence IntervalPreprint
Gutierrez, N. M., & Cribbie, R. A. (2022) · doi:10.31234/osf.io/5buz9 · Cited by 1 · see citing works
da Fonseca, L. O., Mouta, G. D. S., Pinto, A. L. D. C. B., & Resende, G. C. (2025) · Revista Psicologia e Saúde · doi:10.20435/pssa.v1i1.3011
- Authors
- Briones, E. M., & Benham, G.
- Affiliations
- The University of Texas Rio Grande Valley (US)
- Corresponding author
- Benham, Grant
- Venue
- Behavior Research Methods
- Published
- 2016-02-23
- Volume
- 49
- Issue
- 1
- Pages
- 320-334
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2284113891
- Cited by
- 48
- References
- 89
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 5 · 2022: 5 · 2021: 9 · 2020: 6 · 2019: 6 · 2018: 7 · 2017: 5 · 2016: 1
- FWCI
- 19.32
- Type
- book chapter
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- ScholarWorks @ UTRGV (The University of Texas Rio Grande Valley) (submittedVersion)
- Funders
- Center for Online Learning, Teaching and Technology at the University of Texas - Pan American
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- crowdsourcing, online recruitment, Amazon Mechanical Turk, Second Life, self-report measures, psychological stress, sleep quality, online survey tools, undergraduate students
- MeSH terms
- Sleep Hygiene, Adult, Female, Humans, Male, Surveys and Questionnaires, Stress, Psychological, Students, User-Computer Interface, Outcome Assessment, Health Care, Patient Selection, Internet, Self Report, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
