Cited by 1,072Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseEthics in Clinical Research

The DOI matches its Crossref record; 174 of the 177 references it cites were found in a registry or an index.

Conducting Clinical Research Using Crowdsourced Convenience Samples

Chandler, Jesse, Shapiro, Danielle·2016·Annual Review of Clinical Psychology

DOI 10.1146/annurev-clinpsy-021815-093623

Open this paper in CoChat — free account

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

Integrity signals

Registry

Crossref · 10.1146/annurev-clinpsy-021815-093623

Cites

174 of 174 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citation
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citation
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,899
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1×
“In direct comparison studies, MTurk samples are found to be more representative than college student samples, community samples recruited from college towns (Berinsky et al. 2012), or other online sources (Casler et al. 2013).”
Background
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454
Survey Errors and Survey Costs.
Faulkenberry, G. D., & Groves, R. M. (1990) · Journal of the American Statistical Association · doi:10.2307/2289634 · Cited by 1,441
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315
The Body Appreciation Scale-2: Item refinement and psychometric evaluation
Tylka, T. L., & Wood-Barcalow, N. L. (2015) · Body Image · doi:10.1016/j.bodyim.2014.09.006 · Cited by 1,298
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Jeremy Freese, A. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,248
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Self-citation
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
Self-citation
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049
Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“Further, performance on different attention checks in the same survey shows only a modest correlation, and associations between attentiveness items are equally strong regardless of their proximity, suggesting that attention to one question cannot be assumed to guarantee attention to measures administered in close prox…”
Background
Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Self-citation
“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903
Standards for Internet-Based Experimenting
Reips, U. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026/1618-3169.49.4.243 · Cited by 878
Crowdsourcing graphical perception
Heer, J., & Bostock, M. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 819
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798
Demographics of the gay and lesbian population in the United States: Evidence from available systematic data sources
Black, D. A., Gates, G. J., Sanders, S., & Taylor, L. J. (2000) · Demography · doi:10.2307/2648117 · Cited by 777
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772
Intuitive Cooperation and the Social Heuristics Hypothesis: Evidence from 15 Time Constraint Studies
Rand, D. G., Peysakhovich, A., Kraft‐Todd, G., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 772
Beginning the workday yet already depleted? Consequences of late-night smartphone use and sleep
Lanaj, K., Johnson, R. E., & Barnes, C. M. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.01.001 · Cited by 738 · cited 1×
“Consequently, waves of data can be collected from workers over a period of days, allowing for diary studies and other forms of experience sampling (Boynton & Richman 2014, Lanaj et al. 2014, Usinger 2014).”
Background
The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments
Rand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 699
Measuring autistic traits in the general population: a systematic review of the Autism-Spectrum Quotient (AQ) in a nonclinical population sample of 6,900 typical adult males and females
Ruzich, E., Allison, C. L., Smith, P. L., Watson, P., Bonnie Auyeung, B., RING, H. A., & Baron‐Cohen, S. (2015) · Molecular Autism · doi:10.1186/2040-2392-6-2 · Cited by 669
The role of impulsivity in the Dark Triad of personality
Jones, D. N., & Paulhus, D. L. (2011) · Personality and Individual Differences · doi:10.1016/j.paid.2011.04.011 · Cited by 618
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/1807342.1807376 · Cited by 612
Internet-Delivered Psychological Treatments
Andersson, G. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093006 · Cited by 595
Comparing Data Characteristics and Results of an Online Factorial Survey between a Population-Based and a Crowdsource-Recruited Sample
Weinberg, J. D., Jeremy Freese, A., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 482
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458
Personality correlates of the broader autism phenotype as assessed by the Autism Spectrum Quotient (AQ)
Austin, E. J. (2005) · Personality and Individual Differences · doi:10.1016/j.paid.2004.04.022 · Cited by 435
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 360
Collaborative learning in networks
Mason, W., & Watts, D. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1110069108 · Cited by 356
Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review
Ranard, B. L., Ha, Y. P., Meisel, Z. F., Asch, D. A., Hill, S. S., Becker, L. B., Seymour, A. K., & Merchant, R. M. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-013-2536-8 · Cited by 331
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 325
Dimensions of Internet science
Reips, U., & Bošnjak, M. (2001) · Cited by 319
The Association of Parent Mindfulness with Parenting and Youth Psychopathology Across Three Developmental Stages
Parent, J. M., McKee, L. G., Rough, J. N., & Forehand, R. (2015) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-015-9978-x · Cited by 314
The importance of assessing clinical phenomena in Mechanical Turk research.
Hall, K. A. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015) · Psychological Assessment · doi:10.1037/pas0000217 · Cited by 311
The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk
Litman, L., Robinson, J., & Rosenzweig, C. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 310
Twitter versus Facebook: Exploring the role of narcissism in the motives and usage of different social media platforms
Davenport, S. W., Bergman, S. M., Bergman, J. Z., & Fearrington, M. E. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.12.011 · Cited by 305
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 4×
“The number of active users is unknown but is likely close to 15,000 individual US workers at any given time (Stewart et al. 2015; see also Fort et al. 2011), with perhaps a similar number of Indian users (see Indian Workers 56 Chandler · Shapiro A nn u.”
BackgroundResultInfluentialSelf-citation
An MMPI-2 infrequent response scale for use with psychopathological populations: The Infrequency-Psychopathology Scale, F(p).
Arbisi, P. A., & Ben‐Porath, Y. S. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.4.424 · Cited by 304
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data
Benoit, K. R., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016) · American Political Science Review · doi:10.1017/s0003055416000058 · Cited by 296
Being a turker
Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531663 · Cited by 279
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 279
Using the Amazon Mechanical Turk for transcription of spoken language
Marge, M., Banerjee, S., & Rudnicky, A. I. (2010) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2010.5494979 · Cited by 272
Designing Incentives for Inexpert Human Raters
Shaw, A., Horton, J. J., & Chen, D. L. (2010) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1958824.1958865 · Cited by 253
Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 208
Stigmatization of People with Pedophilia: Two Comparative Surveys
Jahnke, S., Imhoff, R., & Hoyer, J. (2014) · Archives of Sexual Behavior · doi:10.1007/s10508-014-0312-4 · Cited by 206
Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults
Blumberg, S. J., & Luke, J. V. (2007) · Public Opinion Quarterly · doi:10.1093/poq/nfm047 · Cited by 192
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
Self-citation
Daily Emotion in Non-Suicidal Self-Injury
Victor, S. E., & Klonsky, E. D. (2013) · Journal of Clinical Psychology · doi:10.1002/jclp.22037 · Cited by 185
Common (Mis)Beliefs about Memory: A Replication and Comparison of Telephone and Mechanical Turk Survey Methods
Simons, D. J., & Chabris, C. F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051876 · Cited by 180
2010 Amendments to the 2002 "Ethical principles of psychologists and code of conduct".
(2010) · American Psychologist · doi:10.1037/a0020168 · Cited by 180
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173
Differential effects of weight bias experiences and internalization on exercise among women with overweight and obesity
Pearl, R. L., Puhl, R. M., & Dovidio, J. F. (2014) · Journal of Health Psychology · doi:10.1177/1359105313520338 · Cited by 166
QRTEngine: An easy solution for running online reaction time experiments using Qualtrics
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0530-7 · Cited by 166
Man enough? Masculine discrepancy stress and intimate partner violence
Reidy, D. E., Berke, D. S., Gentile, B., & Zeichner, A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.04.021 · Cited by 160
The Cyberchondria Severity Scale (CSS): An examination of structure and relations with health anxiety in a community sample
Fergus, T. A. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.05.006 · Cited by 159
Crowdsourcing performance evaluations of user interfaces
Komarov, S., Reinecke, K., & Gajos, K. Z. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470684 · Cited by 157
Ethics and tactics of professional crowdwork
Silberman, M. ‘., Irani, L. C., & Ross, J. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869100 · Cited by 144
Negative affect intensity influences drinking to cope through facets of emotion dysregulation
Veilleux, J. C., Skinner, K. D., Reese, E. D., & Shaver, J. A. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.11.012 · Cited by 138
Professional respondents in nonprobability online panels
Hillygus, D. S., Jackson, N., & Young, M. (2014) · Wiley eBooks · doi:10.1002/9781118763520.ch10 · Cited by 138
Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk
Daly, T. M., & Nataraajan, R. (2015) · Journal of Business Research · doi:10.1016/j.jbusres.2015.05.001 · Cited by 130
Enhancing social relationships through positive psychology activities: a randomised controlled trial
O’Connell, B. H., O’Shea, D., & Gallagher, S. (2016) · The Journal of Positive Psychology · doi:10.1080/17439760.2015.1037860 · Cited by 121
Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 121
Participants at Your Fingertips
Johnson, D. R., & Borden, L. A. (2012) · Teaching of Psychology · doi:10.1177/0098628312456615 · Cited by 119
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Self-citation
Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes
Johnson, P. S., Herrmann, E. S., & Johnson, M. W. (2014) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.110 · Cited by 116
The face of quality in crowdsourcing relevance labels
Kazai, G., Kamps, J., & Milić-Frayling, N. (2012) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2396761.2398697 · Cited by 115
The public stigma of mental illness means a difference between you and me
Corrigan, P. W., Bink, A. B., Fokuo, J. K., & Schmidt, A. (2015) · Psychiatry Research · doi:10.1016/j.psychres.2014.12.047 · Cited by 114
Commonalities in grief responding across bereavement and non-bereavement losses
Papa, A., Lancaster, N. G., & Kahler, J. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.03.018 · Cited by 113
Non-suicidal self-injury disorder in a community sample of adults
Andover, M. S. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.06.001 · Cited by 112
Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility
Schleider, J. L., & Weisz, J. R. (2015) · Journal of Child and Family Studies · doi:10.1007/s10826-015-0126-6 · Cited by 112
Attitudes Over Time: The Psychology of Panel Conditioning
Sturgis, P., Allum, N., & Brunton‐Smith, I. (2009) · Wiley eBooks · doi:10.1002/9780470743874.ch7 · Cited by 110
The Effect of Experimental Mortality on the Internal and External Validity of the Randomized Comparative Experiment
Jurs, S. G., & Glass, G. V. (1971) · The Journal of Experimental Education · doi:10.1080/00220973.1971.11011304 · Cited by 109
LGBT Demographics: Comparisons among population-based surveys
Gates, G. J. (2014) · eScholarship (California Digital Library) · Cited by 109
Evaluating and improving the usability of Mechanical Turk for low-income workers in India
Khanna, S., Ratan, A. L., Davis, J., & Thies, W. (2010) · ACM Symposium on Computing for Development (ACM DEV) · doi:10.1145/1926180.1926195 · Cited by 107
Short Forms of the Social Interaction Anxiety Scale and the Social Phobia Scale
Fergus, T. A., Valentiner, D. P., McGrath, P. B., Gier-Lonsway, S. L., & Kim, H.-S. (2012) · Journal of Personality Assessment · doi:10.1080/00223891.2012.660291 · Cited by 105
The Associations Among Dark Personalities and Sexual Tactics Across Different Scenarios
Jones, D. N., & Olderbak, S. G. (2013) · Journal of Interpersonal Violence · doi:10.1177/0886260513506053 · Cited by 98
Collecting response times using Amazon Mechanical Turk and Adobe Flash
Simcox, T., & Fiez, J. A. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0345-y · Cited by 97
The Five-Factor Narcissism Inventory (FFNI): A test of the convergent, discriminant, and incremental validity of FFNI scores in clinical and community samples.
Miller, J. D., Few, L. R., Wilson, L. F., Gentile, B., Widiger, T. A., MacKillop, J., & Campbell, W. K. (2013) · Psychological Assessment · doi:10.1037/a0032536 · Cited by 96
Online crowdsourcing for efficient rating of speech: A validation study
Byun, T. M., Halpin, P. F., & Szeredi, D. (2014) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2014.11.003 · Cited by 95
Emotion regulation and obsessive–compulsive symptoms: A further examination of associations
Fergus, T. A., & Bardeen, J. R. (2014) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2014.06.001 · Cited by 93
Validation of the Cyberchondria Severity Scale (CSS): Replication and extension with bifactor modeling
Norr, A. M., Allan, N. P., Boffa, J. W., Raines, A. M., & Schmidt, N. B. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.02.001 · Cited by 92
The chronic disease concept of addiction: Helpful or harmful?
Wiens, T. K., & Walker, L. J. (2014) · Addiction Research & Theory · doi:10.3109/16066359.2014.987760 · Cited by 92
A positive effect of flowers rather than eye images in a large-scale, cross-cultural dictator game
Raihani, N., & Bshary, R. (2012) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2012.0758 · Cited by 89
‘Subtypes’ in the Presentation of Autistic Traits in the General Adult Population
Palmer, C. J., Paton, B. K., Enticott, P. G., & Hohwy, J. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2289-1 · Cited by 87
Cognitive behavioral therapy and pharmacotherapy for anxiety: Treatment preferences and credibility among pregnant and non-pregnant women
Arch, J. J. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.11.003 · Cited by 86
Sexual Interest in Children Among an Online Sample of Men and Women
Wurtele, S. K., Simons, D. A., & Moreno, T. (2013) · Sexual Abuse · doi:10.1177/1079063213503688 · Cited by 84
Development and validation of the Specific Loss of Interest and Pleasure Scale (SLIPS)
Winer, E. S., Veilleux, J. C., & Ginger, E. J. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.09.010 · Cited by 83
Turk-Life in India
Gupta, N., Martin, D., Hanrahan, B. V., & O’Neill, J. (2014) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/2660398.2660403 · Cited by 82
Mechanical Turk is Not AnonymousPreprint
Lease, M., Hullman, J. R., Bigham, J. P., Bernstein, M. S., Kim, J., Lasecki, W. S., Bakhshi, S., Mitra, T., & Miller, R. C. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2228728 · Cited by 82
Privacy Attitudes of Mechanical Turk Workers and the U.S. Public
Kang, R., Brown, S. L., Dabbish, L., & Kiesler, S. B. (2014) · Symposium On Usable Privacy and Security · Cited by 78
Support matching and satisfaction in an online breast cancer support community
Vlahovic, T. A., Wang, Y., Kraut, R. E., & Levine, J. M. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557108 · Cited by 77
Lay beliefs about autism spectrum disorder among the general public and childcare providers
Mitchell, G. E., & Locke, K. D. (2015) · Autism · doi:10.1177/1362361314533839 · Cited by 74
Do Mechanical Turks dream of square pie charts?
Kosara, R., & Ziemkiewicz, C. (2010) · doi:10.1145/2110192.2110202 · Cited by 74
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J., Paolacci, G., & Mueller, P. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_30 · Cited by 73
Self-citation
Multidimensional Assessment of Beliefs About Emotion
Veilleux, J. C., Salomaa, A. C., Shaver, J. A., Zielinski, M. J., & Pollert, G. A. (2014) · Assessment · doi:10.1177/1073191114534883 · Cited by 72
Convergent and Discriminant Validity of the Five Factor Form
Rojas, S. L., & Widiger, T. A. (2013) · Assessment · doi:10.1177/1073191113517260 · Cited by 71
Honesty in an Online Labor Market
Suri, S., Goldstein, D. G., & Mason, W. (2013) · Cited by 70
Explaining the relationship between religiousness and substance use: Self-control matters.
DeWall, C. N., Pond, R. S., Carter, E. C., McCullough, M. E., Lambert, N. M., Fincham, F. D., & Nezlek, J. B. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036853 · Cited by 68
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67
Self-citation
An online daily diary study of alcohol use using A mazon's M echanical T urk
Boynton, M. H., & Richman, L. S. (2014) · Drug and Alcohol Review · doi:10.1111/dar.12163 · Cited by 63
Negative and positive urgency may both be risk factors for compulsive buying
Rose, P., & Segrist, D. J. (2014) · Journal of Behavioral Addictions · doi:10.1556/jba.3.2014.011 · Cited by 56
Biological Explanations of Generalized Anxiety Disorder: Effects on Beliefs About Prognosis and Responsibility
Lebowitz, M. S., Pyun, J. J., & Ahn, W. (2013) · Psychiatric Services · doi:10.1176/appi.ps.201300011 · Cited by 51
The credibility of exposure therapy: Does the theoretical rationale matter?
Arch, J. J., Twohig, M. P., Deacon, B. J., Landy, L. N., & Bluett, E. J. (2015) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.05.008 · Cited by 47 · cited 2×
“Some researchers have used MTurk to recruit convenience samples of typically functioning adults to examine public attitudes about clinically relevant issues, such as mental health stigma (Corrigan et al. 2015) and attitudes toward therapy (Arch et al. 2015).”
Background
Psychopathic personality traits predict competitive wins and cooperative losses in negotiation
ten Brinke, L., Black, P., Porter, S., & Carney, D. R. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.02.001 · Cited by 46
Conducting real-time multiplayer experiments on the web
Hawkins, R. X. D. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0515-6 · Cited by 44
Empirical Support for the Moral Salience of the Therapy-Enhancement Distinction in the Debate Over Cognitive, Affective and Social Enhancement
Cabrera, L. Y., Fitz, N. S., & Reiner, P. B. (2014) · Neuroethics · doi:10.1007/s12152-014-9223-2 · Cited by 42
Finding a Comparison Group
Azzam, T., & Jacobson, M. R. (2013) · American Journal of Evaluation · doi:10.1177/1098214013490223 · Cited by 42
Are “Not Just Right Experiences” (NJREs) Specific to Obsessive-Compulsive Symptoms?: Evidence that NJREs Span across Symptoms of Emotional Disorders
Fergus, T. A. (2013) · Journal of Clinical Psychology · doi:10.1002/jclp.22034 · Cited by 37
Online Marketing Research
Miller, J. (2006) · Sage eBooks · doi:10.4135/9781412973380.n7 · Cited by 37
Ethical issues in psychological research on the Internet.
Buchanan, T., & Williams, J. E. (2010) · American Psychological Association eBooks · doi:10.1037/12076-016 · Cited by 36
Hoarding and eating pathology: the mediating role of emotion regulation
Raines, A. M., Boffa, J. W., Allan, N. P., Short, N. A., & Schmidt, N. B. (2015) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2014.11.005 · Cited by 35
Difficulty Identifying Feelings, Distress Tolerance and Compulsive Buying: Analyzing the Associations to Inform Therapeutic Strategies
Rose, P., & Segrist, D. J. (2012) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-012-9389-y · Cited by 35
Who is the Average Individual Investor? Numerical Skills and Implications for Accounting ResearchPreprint
Krische, S. D. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2426570 · Cited by 31
Abstract Analogical Reasoning in High‐Functioning Children with Autism Spectrum Disorders
Green, A. E., Kenworthy, L. E., Mosner, M. G., Gallagher, N. M., Fearon, E. W., Balhana, C. D., & Yerys, B. E. (2014) · Autism Research · doi:10.1002/aur.1411 · Cited by 30
Screening Amazon’s Mechanical Turk for Adults With ADHD
Wymbs, B. T., & Dawson, A. E. (2015) · Journal of Attention Disorders · doi:10.1177/1087054715597471 · Cited by 29
Generating Content for Scenario-Based Serious-Games Using CrowdSourcing
Sina, S., Rosenfeld, A., & Kraus, S. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8790 · Cited by 24
Investigating Several Factors Potentially Related to Endorsing Gambling as an Escape
Weatherly, J. N., & Cookman, M. L. (2014) · Current Psychology · doi:10.1007/s12144-014-9220-y · Cited by 23
Relationships Between Craving Beliefs and Abstinence Self-Efficacy are Mediated by Smoking Motives and Moderated by Nicotine Dependence.
Reese, E., & Veilleux, J. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv054 · Cited by 23
Using Amazon Mechanical Turk to Transcribe Historical Handwritten Documents
Lang, A. S. I. D., & Rio-Ross, J. (2011) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 23
Crowd-powered positive psychological interventions
Morris, R. R., & Picard, R. W. (2014) · The Journal of Positive Psychology · doi:10.1080/17439760.2014.913671 · Cited by 23
Designing a brief measure of social anxiety: Psychometric support for a three-item version of the Interaction Anxiousness Scale (IAS-3)
Nichols, A. L., & Webster, G. D. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.01.043 · Cited by 22
Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge
Cooper, E. A., & Farid, H. (2014) · Public Understanding of Science · doi:10.1177/0963662514554354 · Cited by 21
Uncertainty, god, and scrupulosity: Uncertainty salience and priming god concepts interact to cause greater fears of sin
Fergus, T. A., & Rowatt, W. C. (2014) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2014.09.003 · Cited by 21
Autism-spectrum traits predict humor styles in the general population
Eriksson, K. (2013) · Humor - International Journal of Humor Research · doi:10.1515/humor-2013-0030 · Cited by 21
Human Computation in Citizen Science
Lintott, C. J., & Reed, J. T. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_14 · Cited by 20
Interactive Effects of Anxiety Sensitivity and Emotion Regulation on Anxiety Symptoms
Allan, N. P., Norr, A. M., Macatee, R. J., Gajewska, A., & Schmidt, N. B. (2015) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-015-9484-z · Cited by 19
Establishing the psychometric properties of the Self and Perceived-partner Authenticity in Relationships Scale-Short Form (AIRS-SF): Measurement invariance, reliability, and incremental validity
Wickham, R. E., Reed, D. E., & Williamson, R. E. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2014.12.049 · Cited by 19
Volunteering and Reemployment Status in Unemployed Emerging Adults
Konstam, V., Tomek, S. E., Celen‐Demirtas, S., & Sweeney, K. (2014) · Journal of Career Assessment · doi:10.1177/1069072714523248 · Cited by 19
Age-Related Slowing in Online Samples
Bui, D. C., Myerson, J., & Hale, S. (2015) · The Psychological Record · doi:10.1007/s40732-015-0135-2 · Cited by 18
A Comparison of Six Sample Providers Regarding Online Privacy Benchmarks
Schnorf, S., Sedley, A., Ortlieb, M., & Woodruff, A. (2014) · Cited by 17
Coping with causal uncertainty through alcohol use
Tobin, S. J., Loxton, N. J., & Neighbors, C. (2013) · Addictive Behaviors · doi:10.1016/j.addbeh.2013.11.009 · Cited by 16
Bridging the gap between eye tracking and crowdsourcing
Lebreton, P., Mäki, T., Skodras, E., Hupont, I., & Hirth, M. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2076745 · Cited by 16
Sometimes more competent, but always less warm: Perceptions of biologically oriented mental-health clinicians
Lebowitz, M. S., Ahn, W., & Oltman, K. A. (2015) · International Journal of Social Psychiatry · doi:10.1177/0020764015573086 · Cited by 14
Long-run learning in games of cooperation
Mason, W., Suri, S., & Watts, D. J. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602892 · Cited by 14
Multiple Facets of Problematic Anger Among Regular Smokers: Exploring Associations With Smoking Motives and Cessation Difficulties
Cougle, J. R., Hawkins, K. A., Macatee, R. J., Zvolensky, M. J., & Sarawgi, S. (2014) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu011 · Cited by 13
MTurk Crowdsourcing: A Viable Method for Rapid Discovery of Arabic Nicknames?
Higgins, C., McGrath, E. J., & Moretto, L. (2010) · Cited by 13
Reprint of "Designing a brief measure of social anxiety: Psychometric support for a three-item version of the Interaction Anxiousness Scale (IAS-3)"
Nichols, A., & Webster, G. D. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.03.013 · Cited by 12
Thought Acceleration Boosts Positive Mood Among Individuals with Minimal to Moderate Depressive Symptoms
Yang, K., Friedman‐Wheeler, D. G., & Pronin, E. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-014-9597-9 · Cited by 12
Feeling Good and Taking a Chance? Associations of Hypomania Risk with Cognitive and Behavioral Risk Taking
Devlin, H. C., Johnson, S. L., & Gruber, J. (2015) · Cognitive Therapy and Research · doi:10.1007/s10608-015-9679-3 · Cited by 10
Characteristics of Borderline Personality Disorder and Disgust Sensitivity
Standish, A. J., Benfield, J. A., Bernstein, M. J., & Tragesser, S. L. (2014) · The Psychological Record · doi:10.1007/s40732-014-0039-6 · Cited by 8
Witnessing Disclosure of Depression: Gender and Attachment Avoidance Moderate Interpersonal Evaluations
Burke, S. E., Wang, K., & Dovidio, J. F. (2014) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2014.33.6.536 · Cited by 8
ACT process measures
Gootzeit, J. H. (2014) · doi:10.17077/etd.grjd6zw4 · Cited by 7
Does Stigma Against Smokers Really Motivate Cessation? A Moderated Mediation Model on the Effect of Anti-smoking Campaigns Promoting Smoker-Related Stigma on Cessation Intentions
Kim, J. (2014) · UWM Digital Commons (University of Wisconsin–Milwaukee) · Cited by 6
VARIATIONS IN RECRUITMENT YIELD, COSTS, SPEED AND PARTICIPANT DIVERSITY ACROSS INTERNET PLATFORMS IN A GLOBAL STUDY EXAMINING THE EFFICACY OF AN HIV/AIDS AND HIV TESTING ANIMATED AND LIVE-ACTION VIDEO AMONG ENGLISH- OR SPANISH-SPEAKING INTERNET OR SOCIAL MEDIA USERS.
Shao, W., Guan, W., Clark, M. A., Liu, T., Santelices, C., Cortés, D. E., & Merchant, R. C. (2015) · PubMed · Cited by 6
U.S. residential consumer product information: Validation of methods for post-stratification weighting of Amazon Mechanical Turk surveys
Lawrence Berkeley National Lab. (LBNL), B., Greenblatt, J., USDOE Office of Science (SC), Yang, H.-C., Desroches, L., Young, S., Beraki, B., Price, S., Pratt, S., Willem, H., et al. (2013) · doi:10.2172/1171750 · Cited by 6
An Exploration of Mechanical Turk as a Feasible Recruitment Platform for Cancer Survivors
Carr, A. L. (2014) · CU Scholar (University of Colorado Boulder) · Cited by 5
Pilot study to evaluate the effectiveness of a mobile-based therapy and educational app for children
Howard, A. M., & MacCalla, J. (2014) · doi:10.1145/2668332.2676437 · Cited by 5
Cultural Differences in Playing Repeated Ultimatum Game Online with Virtual Humans
Nouri, E., & Traum, D. R. (2014) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2014.157 · Cited by 4
Cheating on online assessment tests: prevalence and impact on validity
Cavanagh, T. M., Kraiger, K., Gibbons, A., Henry, K., & Maynard, T. (2007) · Dominican Scholar (Dominican University of California) · doi:10.25675/3.024341 · Cited by 4
Shared Sense of Purpose and Well-Being among Veterans and Non-Veterans
Lynn, B. M.-D. (2014) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 4
U.S. Residential Miscellaneous Refrigeration Products: Results from Amazon Mechanical Turk Surveys
Lawrence Berkeley National Lab. (LBNL), B., Greenblatt, J., USDOE Office of Science (SC), Young, S., Yang, H.-C., Long, T., Beraki, B., Price, S., Pratt, S., Willem, H., et al. (2013) · doi:10.2172/1171618 · Cited by 4
Participation in Micro-task Crowdsourcing Markets as Work and Leisure: The Impact of Motivation and Micro-time Structuring
Jiang, L., & Wagner, C. (2014) · Cited by 4
U.S. Residential Miscellaneous Refrigeration Products: Results from Amazon Mechanical Turk Surveys
Greenblatt, J. B., Young, S., Yang, H.-C., Long, T., Beraki, B., Price, S., Pratt, S., Willem, H., & Desroches, L. (2013) · doi:10.2172/1164375 · Cited by 4
Acculturation, Acculturative Stress, Social Status, and Well-Being among English Language Proficient Immigrants
Bernal, D. R. (2014) · Cited by 3
Using the Crowd to Generate Content for Scenario-Based Serious-GamesPreprint
Sina, S., Kraus, S., & Rosenfeld, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.5034 · Cited by 3
Effect of Internet Administered Mindfulness Training on Anxiety and Sleep Quality
Usinger, T. (2014) · CU Scholar (University of Colorado Boulder) · Cited by 1

Research works printed but not linked: 12

Barnhoorn JS, 2014, Behav. Res. Methods, 530, 1
Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
Downes-Le Guin T, 2006, ESOMAR World Res. Conf. Panel Res.
Higgins C, 2010, Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, 89
10.1007/s10683-011-9273-9 · DOI
10.1145/2676431.2676437 · DOI
Lang ASID, 2011, Code4Lib J., 15, 1
Lebreton P, 2015, Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco
10.1026//1618-3169.49.4.243 · DOI
Shao W, 2015, Digit. Cult. Educ., 7
Suri S, 2011, AAAI Workshops, 25th, San Francisco, 61

Sorted by citations

Understanding and managing the threat of common method bias: Detection, prevention and control
Kock, F., Berbekova, A., & Assaf, A. G. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104330 · Cited by 1,692
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 815
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Self-citation
Customer service chatbots: Anthropomorphism and adoption
Sheehan, B., Jin, H. S., & Gottlieb, U. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.04.030 · Cited by 667
Mental health during the COVID-19 pandemic: Effects of stay-at-home policies, social distancing behavior, and social resources
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 667

See all 1,072

Authors: Chandler, J., & Shapiro, D.
Affiliations: Mathematica Policy Research (US)
University of Michigan (US)
ORCID: Chandler, Jesse
Venue: Annual Review of Clinical Psychology
Publisher: Annual Reviews
Published: 2016-03-28
Volume: 12
Issue: 1
Pages: 53-81
ISSN: 1548-5943 (ISSN-L)
1548-5951
DOI: 10.1146/annurev-clinpsy-021815-093623
OpenAlex: W2126493218
Cited by: 1,072
References: 177
Citations by year: 2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
FWCI: 17.58
Type: journal article
Language: en
Open access: closed
Locations: Annual Review of Clinical Psychology (publishedVersion)
PubMed (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
Keywords: crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
MeSH terms: Humans, Biomedical Research, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Conducting Clinical Research Using Crowdsourced Convenience Samples

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks