Comparison of Online Survey Recruitment Platforms for Hard-to-Reach Pregnant Smoking Populations: Feasibility Study
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Recruiting hard-to-reach populations for health research is challenging. Web-based platforms offer one way to recruit specific samples for research purposes, but little is known about the feasibility of online recruitment and the representativeness and comparability of samples recruited through different Web-based platforms. OBJECTIVE: The objectives of this study were to determine the feasibility of recruiting a hard-to-reach population (pregnant smokers) using 4 different Web-based platforms and to compare participants recruited through each platform. METHODS: A screener and survey were distributed online through Qualtrics Panel, Soapbox Sample, Reddit, and Amazon Mechanical Turk (mTurk). Descriptive statistics were used to summarize results of each recruitment platform, including eligibility yield, quality yield, income, race, age, and gestational age. RESULTS: Of the 3847 participants screened for eligibility across all 4 Web-based platforms, 535 were eligible and 308 completed the survey. Amazon mTurk yielded the fewest completed responses (n=9), 100% (9/9) of which passed several quality metrics verifying pregnancy and smoking status. Qualtrics Panel yielded 14 completed responses, 86% (12/14) of which passed the quality screening. Soapbox Sample produced 107 completed surveys, 67% (72/107) of which were found to be quality responses. Advertising through Reddit produced the highest completion rate (n=178), but only 29.2% (52/178) of those surveys passed the quality metrics. We found significant differences in eligibility yield, quality yield, age, number of previous pregnancies, age of smoking initiation, current smokers, race, education, and income (P<.001). CONCLUSIONS: Although each platform successfully recruited pregnant smokers, results varied in quality, cost, and percentage of complete responses. Moving forward, investigators should pay careful attention to the percentage yield and cost of online recruitment platforms to maximize internal and external validity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 3×
“On mTurk, “spammers” and “bots” capitalize on this and find ways to receive rewards offered by a study without successfully completing the intended task [18,30] with obvious adverse consequences for the data validity.”
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 1×
“As is typical in Web-based survey research, we employed multiple attention checks and quality screens in our surveys [23].”
STRUCTURAL RACISM AND HEALTH INEQUITIES: Old Issues, New Directions
Gee, G. C., & Ford, C. L. (2011) · Du Bois Review: Social Science Research on Race · doi:10.1017/s1742058x11000130 · Cited by 1,388
Understanding the potential of teachable moments: the case of smoking cessation
McBride, C. M. (2003) · Health Education Research · doi:10.1093/her/18.2.156 · Cited by 1,162 · cited 1×
“pregnancy than any other time in their lives [10,11], only one-third of women are able to remain abstinent [12].”
Interventions for promoting smoking cessation during pregnancy
Lumley, J., Chamberlain, C., Dowswell, T., Oliver, S., Oakley, L., & Watson, L. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001055.pub3 · Cited by 890
Stigma, shame, and blame experienced by patients with lung cancer: qualitative study
Chapple, A., Ziebland, S., & McPherson, A. (2004) · BMJ · doi:10.1136/bmj.38111.639734.7c · Cited by 789 · cited 1×
“Furthermore, biologically confirmed smoking status (eg, cotinine testing) is expensive [15,16].”
Ramo, D. E., & Prochaska, J. J. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1878 · Cited by 404
Briones, E. M., & Benham, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0710-8 · Cited by 48 · cited 1×
“Crowdsourcing platforms such as Amazon’s Mechanical Turk (mTurk), Soapbox Sample, and Qualtrics Panel Data draw a broad demographic of workers that can meet even very specific and targeted inclusion criteria [18,19].”
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Shensa, A., Sidani, J. E., Escobar-Viera, C. G., Switzer, G. E., Primack, B. A., & Choukas-Bradley, S. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.08.092 · Cited by 129
“Qualtrics Sampling Service is a subdivision of Qualtrics, a private research software company specializing in Web-based data collection that partners with over 20 Web-based panel providers to supply diverse, quality respondents (Ibarra et al., 2018).”
BackgroundMindfulness and Depersonalization: a Nuanced Relationship
Levin, K. K., Gornish, A., & Quigley, L. (2022) · Mindfulness · doi:10.1007/s12671-022-01890-y · Cited by 15
“Moreover, studies support the representativeness and quality of data collected through Qualtrics over other online platforms (Boas et al., 2020; Heen et al., 2014; Ibarra et al., 2018).”
BackgroundWhy do cognitive prompts hurt learning in older adults?
Kraiger, K., Cavanagh, T. M., & Willis, C. M. G. (2020) · International Journal of Training and Development · doi:10.1111/ijtd.12169 · Cited by 12
Metadata
- Authors
- Ibarra, J. L., Agas, J. M., Lee, M., Pan, J. L., & Buttenheim, A. M.
- Venue
- JMIR Research Protocols
- Publisher
- JMIR Publications Inc.
- Published
- 2018-04-16
- Volume
- 7
- Issue
- 4
- Pages
- e101
- ISSN
- 1929-0748 (ISSN-L)
- OpenAlex
- W2800811210
- Cited by
- 87
- References
- 9
- FWCI
- 21.64
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.