Selection Bias in Web Surveys
Open accessAbstract
Summary: At first sight, web surveys seem to be an interesting and attractive means of data collection. They provide simple, cheap, and fast access to a large group of potential respondents. However, web surveys are not without methodological problems. Specific groups in the populations are under‐represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self‐selection. Both under‐coverage and self‐selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under‐coverage problems may solve itself in the future, but that self‐selection leads to unreliable survey outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reliability and validity of web-based self-reported mental disorder symptom clusters remains ambiguous, certainly for the current population.
“The reliability and validity of web-based self-reported mental disorder symptom clusters remains ambiguous,71 certainly for the current population”
From Mental Disorder Symptoms among Public Safety Personnel in Canada (2018) · cites this paper as [71]
The paper discusses measurement errors in web surveys and warns in §5.3 (page not supplied) that under-coverage and self-selection can bias population estimates. It does not assess the reliability or validity of self-reported mental-disorder symptom clusters, including in the citing paper’s population. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe current rates may be inflated because of a self-selection bias of motivated responding by clinically symptomatic persons.
“In contrast, the current rates may be inflated because of a self-selection bias of motivated responding by clinically symptomatic persons.71”
From Mental Disorder Symptoms among Public Safety Personnel in Canada (2018) · cites this paper as [71]
Section 2.3 (page unavailable) explains that self-selection can bias survey estimates upward when people with higher values of the measured characteristic are more likely to respond. It does not examine clinically symptomatic people or establish that they were more motivated to respond in the citing study. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008) · Cited by 5,354 · see citing works
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293 · see citing works
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082 · see citing works
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011 · see citing works
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,086 · see citing works
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086 · see citing works
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961 · see citing works
Neyman, J. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_12 · Cited by 932 · see citing works
To mix or not to mix data collection modes in surveys.
de Leeuw, E. D. (2005) · Journal of Official Statistics · Cited by 808 · see citing works
Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature
Fricker, R. D., & Schonlau, M. (2002) · Field Methods · doi:10.1177/152582202237725 · Cited by 747 · see citing works
Conducting Research Surveys via E-mail and the Web
Schonlau, M., Fricker, R. D., & Elliott, M. N. (2002) · RAND Corporation eBooks · doi:10.7249/mr1480 · Cited by 603 · see citing works
Comparing Data from Online and Face-to-face Surveys
Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005) · International Journal of Market Research · doi:10.1177/147078530504700602 · Cited by 569 · see citing works
Estimation in Surveys with Nonresponse
Särndal, C., & Lundström, S. (2005) · Wiley eBooks · doi:10.1002/0470011351 · Cited by 499 · see citing works
Designing Effective Web Surveys
Couper, M. P. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511499371 · Cited by 472 · see citing works
Sampling Methods for Censuses and Surveys.
Deming, W. E., & Yates, F. (1949) · Journal of the American Statistical Association · doi:10.2307/2279916 · Cited by 443 · see citing works
Dimensions of Internet science
Reips, U., & Bošnjak, M. (2001) · Cited by 319 · see citing works
Deming, W. E. (2010) · Andalas University Repository (Andalas University) · Cited by 292 · see citing works
Computer Assisted Survey Information Collection
(2000) · Technometrics · doi:10.1080/00401706.2000.10486039 · Cited by 202 · see citing works
C., A., & Deming, W. E. (1950) · Population · doi:10.2307/1523702 · Cited by 199 · see citing works
Representative Sampling, III: The Current Statistical Literature
Kruskal, W., & Mosteller, F. (1979) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1402647 · Cited by 194 · see citing works
Applied Survey Methods: A Statistical Perspective
Bethlehem, J. G. (2009) · Medical Entomology and Zoology · Cited by 187 · see citing works
Self-citationBethlehem, J. G. (2009) · Wiley eBooks · doi:10.1002/9780470494998 · Cited by 182 · see citing works
Self-citationA Comparison Between Responses From a Propensity-Weighted Web Survey and an Identical RDD Survey
Schonlau, M., Zapert, K., Simon, L. P., Haynes-Sanstad, K., Marcus, S. M., Adams, J., Spranca, M. D., Kan, H., Turner, R., & Berry, S. H. (2004) · Social Science Computer Review · doi:10.1177/0894439303256551 · Cited by 171 · see citing works
Weighting nonresponse adjustments based on auxiliary information
Bethlehem, J. G. (2002) · UvA-DARE (University of Amsterdam) · Cited by 143 · see citing works
Self-citationRepresentative Sampling, II: Scientific Literature, Excluding Statistics
Kruskal, W., & Mosteller, F. C. (1979) · International Statistical Review · doi:10.2307/1402564 · Cited by 141 · see citing works
Representative Sampling, I: Non-Scientific Literature
Kruskal, W., & Mosteller, F. C. (1979) · International Statistical Review · doi:10.2307/1403202 · Cited by 131 · see citing works
An Evaluation of the Effect of Response Formats on Data Quality in Web Surveys
Heerwegh, D., & Loosveldt, G. (2002) · Social Science Computer Review · doi:10.1177/089443902237323 · Cited by 89 · see citing works
Computer Assisted Survey Information Collection.
Morganstein, D. R. (1991) · Journal of the American Statistical Association · doi:10.2307/2290617 · Cited by 87 · see citing works
How Representative Are Online Panels? Problems of Coverage and Selection and Possible Solutions
Scherpenzeel, A. C. (2018) · Routledge eBooks · doi:10.4324/9780203844922-5 · Cited by 64 · see citing works
Bowley, A. L. (1906) · Journal Of The Royal Statistical Society · doi:10.2307/2339344 · Cited by 59 · see citing works
Access panels and online research, panacea or pitfall?
Stoop, I., & Wittenberg, M. (2008) · Data Archiving and Networked Services (DANS) · Cited by 20 · see citing works
Kendall, M. G. (1960) · Biometrika · doi:10.1093/biomet/47.3-4.447 · Cited by 18 · see citing works
Reducing the bias of web survey based estimates
Bethlehem, J. G. (2007) · Cited by 15 · see citing works
Self-citationAddress to the economic science and statistics section of the British Association
Giffen, R. (1887) · Printed by Spottiswoode eBooks · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Cochran, W.G. (1953) Sampling Techniques. New York: John Wiley & Sons.
- Cochran, W.G. (1977). Sampling Techniques, 3rd ed. New York: John Wiley & Sons.
- Couper, M.P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494.
- Hansen, M.H., Hurvitz, W.N. & Madow, W.G. (1953). Survey Sampling Methods and Theory. New York: John Wiley & Sons.
- Kish, L. (1967). Survey Sampling. New York: Wiley & Sons.
- Kuusela, V. (2003). Mobile phones and telephone survey methods. In Proceedings of the 4th ASC International Conference, Eds. R. Banks, J. Currall, J. Francis, L. Gerrard, R. Kahn, T. Macer, M. Rigg, E. Ross, S. Taylor & A. Westlake, pp. 317-327. Chesham Bucks, UK: Association for Survey Computing (ASC).
- C 2010 The Author. Journal compilation C 2010 International Statistical Institute
Cited by
Sorted by citations
Expectations of brilliance underlie gender distributions across academic disciplines
Leslie, S.-J., Cimpian, A., Meyer, M., & Freeland, E. (2015) · Science · doi:10.1126/science.1261375 · Cited by 1,516 · see citing works
Mental Disorder Symptoms among Public Safety Personnel in Canada
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Duranceau, S., LeBouthillier, D. M., Sareen, J., Ricciardelli, R., MacPhee, R. S., Groll, D., et al. (2018) · The Canadian Journal of Psychiatry · doi:10.1177/0706743717723825 · Cited by 558 · see citing works
“In contrast, the current rates may be inflated because of a self-selection bias of motivated responding by clinically symptomatic persons.(71) That argument might be countered by notions that particularly symptomatic persons may also be less likely to participate in research.”
BackgroundThe role of urban green space for human well-being
Bertram, C., & Katrin Rehdanz, A. (2015) · Ecological Economics · doi:10.1016/j.ecolecon.2015.10.013 · Cited by 515 · see citing works
Associations between green/blue spaces and mental health across 18 countries
White, M. P., Elliott, L. R., Grellier, J., Economou, T., Bell, S., Bratman, G. N., Cirach, M., Gascón, M., Lima, M. L., Lõhmus, M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-87675-0 · Cited by 503 · see citing works
Krieger, N. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094017 · Cited by 411 · see citing works
“, mainly highly educated and affluent persons) (18, 22, 201).”
BackgroundSocial Media, Web, and Panel Surveys: Using Non‐Probability Samples in Social and Policy Research
Lehdonvirta, V., Oksanen, A., Räsänen, P., & Blank, G. (2020) · Policy & Internet · doi:10.1002/poi3.238 · Cited by 394 · see citing works
Bono, S. A., Villela, E. F. D. M., Siau, C. S., Chen, W. S., Pengpid, S., Hasan, M. T., Sessou, P., Ditekemena, J., Amodan, B. O., Hosseinipour, M. C., et al. (2021) · Vaccines · doi:10.3390/vaccines9050515 · Cited by 350 · see citing works
“Therefore, weights were applied to the study data [48].”
MethodInference for Nonprobability Samples
Elliott, M. R., & Valliant, R. (2017) · Statistical Science · doi:10.1214/16-sts598 · Cited by 331 · see citing works
“Bethlehem (2010) reviews this issue for web surveys. Vonk, van Ossenbruggen and Willems (2006) report that ethnic minorities and immigrant groups were systematically underrepresented in Dutch panels. They also found that, relative to the general population, the Dutch online panels contained disproportionately more vot…”
BackgroundInfluentialWeigold, A., Weigold, I. K., & Russell, E. J. (2013) · Psychological Methods · doi:10.1037/a0031607 · Cited by 333 · see citing works
Hoffkling, A., Obedin‐Maliver, J., & Sevelius, J. M. (2017) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-017-1491-5 · Cited by 311 · see citing works
“As other studies have shown, participants engaging in online studies are disproportionately educated and economically secure [24].”
ResultThe use of sampling methods in advertising research: a gap between theory and practice
Sarstedt, M., Bengart, P., Shaltoni, A. M., & Lehmann, S. (2017) · International Journal of Advertising · doi:10.1080/02650487.2017.1348329 · Cited by 294 · see citing works
New Developments in Survey Data Collection
Couper, M. P. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053613 · Cited by 253 · see citing works
Busch, I. M., Moretti, F., Purgato, M., Barbui, C., Wu, A. W., & Rimondini, M. (2019) · Journal of Patient Safety · doi:10.1097/pts.0000000000000589 · Cited by 231 · see citing works
Service quality in a higher education context: an integrated model
Sultan, P., & Wong, H. Y. (2012) · Asia Pacific Journal of Marketing and Logistics · doi:10.1108/13555851211278196 · Cited by 238 · see citing works
“Thus, a random technique was adopted (Bryman and Bell, 2007; Bethlehem, 2010).”
MethodKhazaal, Y., van Singer, M., Chatton, A., Achab, S., Zullino, D., Rothen, S., Khan, R., Billieux, J., & Thorens, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2759 · Cited by 201 · see citing works
“It is difficult to estimate the impact of any selection bias because information on non-participants is usually not available, and comparisons between the included and the excluded samples are not feasible [12].”
BackgroundResultInfluentialRadojević, T., Stanišić, N., & Stanić, N. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.04.002 · Cited by 194 · see citing works
What’s to Like? Facebook as a Tool for Survey Data Collection
Schneider, D., & Harknett, K. (2019) · Sociological Methods & Research · doi:10.1177/0049124119882477 · Cited by 192 · see citing works
“The weight of early research and discussion suggested that problems of undercoverage from limited and selective Internet usage made this approach of limited use (Best, Krueger, and Hubbard 2001; Bethlehem 2010; Yeager et al. 2011).”
BackgroundPerils and potentials of self‐selected entry to epidemiological studies and surveys
Keiding, N., & Louis, T. A. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12136 · Cited by 199 · see citing works
Kelly, A., Sullivan, M., & Strampel, K. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.6.12 · Cited by 177 · see citing works
“Finally, the authors acknowledge the general limitations of online surveys with respect to self-selection bias and potentially limited internet access of target participants during the time in which the survey was conducted (Bethlehem, 2010).”
Impacts of Covid-19 on peer-to-peer accommodation platforms: Host perceptions and responses
Farmaki, A., Miguel, C., Drotarova, M. H., Aleksić, A., Časni, A. Č., & Efthymiadou, F. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102663 · Cited by 163 · see citing works
“In order to minimise self-selection bias (Bethlehem, 2010), the researchers tried to ensure that the sample from each country was diverse enough (Ritchie et al., 2014) in terms of gender, age and host type.”
Estimating Propensity Adjustments for Volunteer Web Surveys
Valliant, R. L., & Dever, J. A. (2011) · Sociological Methods & Research · doi:10.1177/0049124110392533 · Cited by 167 · see citing works
“The model can be a simple one where means differ among groups, leading to the poststra-tified estimator (e.g., see Bethlehem 2010), or a more elaborate one involving multiple covariates, as used by Dever, Rafferty, and Valliant (2008).”
MethodCurrie, G., Jain, T., & Aston, L. (2021) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2021.09.009 · Cited by 164 · see citing works
Russo, D., Hanel, P. H. P., Altnickel, S., & van Berkel, N. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09945-9 · Cited by 168 · see citing works
“In sum, it is a convenient, fair, and efficient way to recruit survey informants (Bethlehem 2010).”
BackgroundLiu, Y., Huang, K., Bao, J., & Chen, K. (2018) · Tourism Management · doi:10.1016/j.tourman.2018.10.004 · Cited by 162 · see citing works
What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 166 · see citing works
“In our survey, self-selection bias may have also influenced our results [3]: our respondents may feel more strongly about pain points in notebooks than the general population of data scientists.”
ResultThe Importance of Selection Bias in Internet Surveys
Greenacre, Z. A. (2016) · Open Journal of Statistics · doi:10.4236/ojs.2016.63035 · Cited by 147 · see citing works
Suicidal ideation, plans, and attempts among public safety personnel in Canada
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., LeBouthillier, D. M., Duranceau, S., Sareen, J., Ricciardelli, R., MacPhee, R. S., Groll, D., et al. (2018) · Canadian Psychology / Psychologie canadienne · doi:10.1037/cap0000136 · Cited by 151 · see citing works
“Relatedly, the reliability and validity of Web based self-reports relative to clinician administered interviews remains ambiguous (Bethlehem, 2010).”
BackgroundReport of the AAPOR task force on non-probability sampling
Baker, R., Brick, J. M., Bates, N., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Cited by 151 · see citing works
Mental health training, attitudes toward support, and screening positive for mental disorders
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Vaughan, A. D., Anderson, G. S., Ricciardelli, R., MacPhee, R. S., Cramm, H. A., Czarnuch, S., et al. (2020) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2019.1575900 · Cited by 135 · see citing works
Patterns of Implicit and Explicit Attitudes: IV. Change and Stability From 2007 to 2020
Charlesworth, T. E. S., & Banaji, M. R. (2022) · Psychological Science · doi:10.1177/09567976221084257 · Cited by 138 · see citing works
“After all, as a convenience sample, the data set can suffer from selection bias (Bethlehem, 2010).”
BackgroundSuspicious and fraudulent online survey participation: Introducing the REAL framework
Lawlor, J., Thomas, C. C., Guhin, A. T., Kenyon, K., Lerner, M. D., & Drahota, A. (2021) · Methodological Innovations · doi:10.1177/20597991211050467 · Cited by 142 · see citing works
“…include self-selection bias among participants, under-coverage of individuals without Internet access, and participants’ ability to misrepre-sent their eligibility to participate or mask their identities in order to participate multiple times in a single study (Bethlehem, 2010; Wright, 2005).”
Remote smartphone monitoring of Parkinson’s disease and individual response to therapy
Omberg, L., Neto, E. C., Perumal, T. M., Pratap, A., Tediarjo, A., Adams, J., Bloem, B. R., Bot, B. M., Elson, M. J., Goldman, S. M., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00974-9 · Cited by 140 · see citing works
Fibromyalgia: a new facet of the post-COVID-19 syndrome spectrum? Results from a web-based survey
Ursini, F., Ciaffi, J., Mancarella, L., Lisi, L., Brusi, V., Cavallari, C., D’Onghia, M., Mari, A., Borlandelli, E., Cordella, J. F., et al. (2021) · RMD Open · doi:10.1136/rmdopen-2021-001735 · Cited by 135 · see citing works
Krouwel, A., Kutiyski, Y., van Prooijen, J.-W., Martinsson, J., & Markstedt, E. (2017) · Journal of Social and Political Psychology · doi:10.5964/jspp.v5i2.745 · Cited by 137 · see citing works
“Datasets collected online generally suffer from problems of under-coverage and self-selection that can potentially bias estimates (Bethlehem, 2010; Hooghe & Teepe, 2007).”
Meule, A., Müller, A., Gearhardt, A. N., & Jens Blechert, A. (2016) · Appetite · doi:10.1016/j.appet.2016.10.003 · Cited by 136 · see citing works
Influencing factors for the purchase intention of consumers choosing bioplastic products in Germany
Klein, F., Emberger‐Klein, A., Menrad, K., Möhring, W., & Blesin, J.-M. (2019) · Sustainable Production and Consumption · doi:10.1016/j.spc.2019.01.004 · Cited by 136 · see citing works
How the European debt crisis reshaped national political space: The case of Greece
Κατσανίδου, Α., & Otjes, S. (2015) · European Union Politics · doi:10.1177/1465116515616196 · Cited by 135 · see citing works
“Like most online surveys, VAA samples suffer from undercoverage and self-selection that could potentially bias estimates (Bethlehem, 2010).”
BackgroundBruckner, S., Wilson, M., Aurell, D., Rennich, K., vanEngelsdorp, D., Steinhauer, N., & Williams, G. R. (2023) · Journal of Apicultural Research · doi:10.1080/00218839.2022.2158586 · Cited by 134 · see citing works
“…employ different data collection and reporting approaches which harbor their own advantages and disadvantages (Evans & Mathur, 2005); for example, while online questionnaires provide the benefit of a wide geographical reach, they disadvantage beekeepers without internet access (Bethlehem, 2010).”
BackgroundInformation Could Reduce Consumer Demand for Exotic Pets
Moorhouse, T. P., Balaskas, M., D’Cruze, N. C., & Macdonald, D. W. (2016) · Conservation Letters · doi:10.1111/conl.12270 · Cited by 133 · see citing works
“Our study was limited to respondents with internet access (Bethlehem 2010) and whose Web browsers searched for English language websites.”
BackgroundInfluentialA multi-group analysis of convenience samples: free, cheap, friendly, and fancy sources
Winton, B. G., & Sabol, M. A. (2021) · International Journal of Social Research Methodology · doi:10.1080/13645579.2021.1961187 · Cited by 143 · see citing works
“We understand issues of non-random selection and assignment inherent with the use of either traditional or online convenience sampling techniques (Bethlehem, 2010; Shadish et al., 2002).”
BackgroundMethodStatistical Challenges of Administrative and Transaction Data
Hand, D. J. (2018) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12315 · Cited by 129 · see citing works
“De Veaux and Hand (2005) gave examples of data errors and their consequences, and national statistical institutes often define several dimensions of quality, including accuracy, relevance, timeliness, existence, coherence, completeness, accessibility and security (see, for example, Eurostat (2000) and the archives of…”
BackgroundPsychometric Properties of the Greek Version of FCV-19S
Tsipropoulou, V., Nikopoulou, V. A., Holeva, V., Nassika, Z., Diakogiannis, I., Sakka, S., Kostikidou, S., Varvara, C., Spyridopoulou, E., & Parlapani, E. (2020) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-020-00319-8 · Cited by 127 · see citing works
Morse, K. F., Fine, P. A., & Friedlander, K. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.609967 · Cited by 127 · see citing works
“As the survey was administered online, response rates were likely affected by self-selection biases, that is, completed by those interested in leisure during COVID-19 or who had additional time to do so (Bethlehem, 2010).”
MethodKawamura, T., Mori, T., Motonishi, T., & Ogawa, K. (2021) · Journal of the Japanese and International Economies · doi:10.1016/j.jjie.2021.101131 · Cited by 147 · see citing works
Kim, B., & Kim, M. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100590 · Cited by 137 · see citing works
Who mothers mommy? Factors that contribute to mothers’ well-being.
Luthar, S. S., & Ciciolla, L. M. (2015) · Developmental Psychology · doi:10.1037/dev0000051 · Cited by 117 · see citing works
“Limitations and caveats Given our web-based data collection, there remains a possibility of some sampling biases (Bethlehem, 2010), but at least one major source of such bias – lack of access to computers – is unlikely to have been a major problem given our focus on upper middle class mothers.”
BackgroundSurvey Weights: A Step-by-step Guide to Calculation
Dever, J. A., & Valliant, R. L. (2017) · Cited by 116 · see citing works
Pettinicchio, D., Maroto, M. L., Chai, L., & Lukk, M. (2021) · Disability and health journal · doi:10.1016/j.dhjo.2021.101085 · Cited by 114 · see citing works
COVID-19 Vaccination Acceptance and Hesitancy among Healthcare Workers in Germany
Holzmann‐Littig, C., Braunisch, M. C., Kranke, P., Popp, M., Seeber, C., Fichtner, F., Littig, B., Carbajo-Lozoya, J., Allwang, C., Frank, T., et al. (2021) · Vaccines · doi:10.3390/vaccines9070777 · Cited by 114 · see citing works
“Selection bias is a common problem in online surveys [58], which cannot be ruled out.”
BackgroundGoing web or staying paper? The use of web-surveys among older people
Kelfve, S., Kivi, M., Johansson, B. E. A., & Lindwall, M. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01138-0 · Cited by 114 · see citing works
Cheah, J., Roldán, J. L., Ciavolino, E., Ting, H., & Ramayah, T. (2020) · Total Quality Management & Business Excellence · doi:10.1080/14783363.2020.1754125 · Cited by 111 · see citing works
“Finally, researchers should engage in different ex-post adjustments of sampling unit weights to compensate for critical issues in survey research, such as unequal selection probabilities, non-coverage, and sampling fluctuations (Bethlehem, 2010; Sarstedt et al., 2018a).”
BackgroundTobacco Use Patterns in Five Countries During the COVID-19 Lockdown
Yach, D. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa097 · Cited by 105 · see citing works
Mental health in chronic disease patients during the COVID-19 quarantine in Greece
Louvardi, M., Pelekasis, P., Chrousos, G. P., & Darviri, C. (2020) · Palliative & Supportive Care · doi:10.1017/s1478951520000528 · Cited by 105 · see citing works
Humer, E. C., Stippl, P., Pieh, C., Pryss, R. C., & Probst, T. (2020) · Journal of Medical Internet Research · doi:10.2196/20246 · Cited by 108 · see citing works
“The web-based conduction of the survey may also have caused a selection bias toward participation by fewer older psychotherapists [40].”
BackgroundRestar, A. (2019) · Archives of Sexual Behavior · doi:10.1007/s10508-019-1453-2 · Cited by 103 · see citing works
“…is a type of bias that, when introduced in the survey, motivates participants (i.e., parental-respondents) to elect themselves into the study such that this selection is different from the study enrollment criteria (Bethlehem, 2010; Cozby, 2012; DelgadoRodriguez & Llorca, 2004; Ruane, 2005).”
ResultVannini, P., Gagliardi, G. P., Kuppe, M. K., Dossett, M. L., Donovan, N. J., Gatchel, J. R., Quiroz, Y. T., Premnath, P. Y., Amariglio, R. E., Sperling, R. A., et al. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.03.050 · Cited by 107 · see citing works
“We acknowledge that specific groups in the population may be under-represented as they may not have an email address and/or have limited or no access to internet (Bethlehem 2010).”
BackgroundInfluentialMobile Research Methods: Opportunities and challenges of mobile research methodologies
Toninelli, D., Pintér, R., & de Pedraza, P. (2015) · Ubiquity Press eBooks · doi:10.5334/bar · Cited by 98 · see citing works
The power of storytelling and video: a visual rhetoric for science communication
Finkler, W., & León, B. (2019) · Journal of Science Communication · doi:10.22323/2.18050202 · Cited by 105 · see citing works
“The research design had some limitations that should be kept in perspective when considering the results: (i) while the online survey provided a cost- and time-effective means of garnering a large sample size, the online and social media nature of its distribution introduced a self-selection bias, whereby survey compl…”
BackgroundResultWhat it feels like to be a mother: Variations by children’s developmental stages.
Luthar, S. S., & Ciciolla, L. M. (2015) · Developmental Psychology · doi:10.1037/dev0000062 · Cited by 95 · see citing works
“There remains a possibility of some sampling biases given our web-based data-collection (Bethlehem, 2010), but at least one major source of such bias – lack of access to computers – is unlikely to have been a serious problem given our targeted sample of upper middle class mothers.”
BackgroundOptions for Conducting Web Surveys
Schonlau, M., & Couper, M. P. (2017) · Statistical Science · doi:10.1214/16-sts597 · Cited by 95 · see citing works
“Bias and variance are reduced if both conditions hold (Bethlehem, 2010).”
BackgroundMondon, A. (2022) · Politics · doi:10.1177/02633957221104726 · Cited by 94 · see citing works
“…crucial to note that my interest here is not to engage with methodological issues regarding the collection of data, which have been well covered already (see, among others, Bernstein et al., 2001; Bethlehem, 2010; Bon et al., 2018; Harbaugh, 1996; Karp and Brockington, 2005; Morton et al., 2015).”
Heijnen, M., & Rietdyk, S. (2015) · Human Movement Science · doi:10.1016/j.humov.2015.12.007 · Cited by 93 · see citing works
Warren, A. M., Zolfaghari, K., Fresnedo, M., Bennett, M., Pogue, J. R., Waddimba, A. C., Zvolensky, M. J., Carlbring, P., & Powers, M. B. (2021) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2021.1874505 · Cited by 89 · see citing works
“Although our sample size was large, we relied on the use of Qualtrics patient panels thus the sample may have some bias towards individuals who self-select to be participants for online survey research (Bethlehem, 2010; Bhole & Hanna, 2017; Greenacre, 2016; Søgaard et al., 2004).”
MethodZhao, H., He, X., Fan, G., Li, L., Huang, Q., Qiu, Q.-M., Kang, Z., Du, T., Han, L., Ding, L., et al. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.085 · Cited by 88 · see citing works
“Therefore, this survey may suffer from the problems of under-coverage and selfselection (Bethlehem, 2010; Greenacre, 2016).”
Psychological distress and coping styles in teachers: A preliminary study
Stapleton, P., Garby, S., & Sabot, D. (2020) · Australian Journal of Education · doi:10.1177/0004944120908960 · Cited by 92 · see citing works
“A second limitation involves the self-selection of the convenience sample, which can lead to biased estimates (Bethlehem, 2010).”
Moeller, J., Brackett, M. A., Ivčević, Z., & White, A. E. (2020) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.101301 · Cited by 93 · see citing works
Hwang, J., Kim, I., & Gulzar, M. A. (2020) · Sustainability · doi:10.3390/su12041440 · Cited by 84 · see citing works
“In particular, as a cross-cultural testing is very important for environmental research [54] it is necessary to verify the research model proposed in this study based on the cross-cultural test in future studies; Second, this paper focused on the context of DFDS, so the results of this study could be different if the…”
MethodDemartini, E., Vecchiato, D., Tempesta, T., Gaviglio, A., & Viganò, R. (2018) · Meat Science · doi:10.1016/j.meatsci.2018.07.031 · Cited by 83 · see citing works
“The authors acknowledge that this might be a limitation of the study and might introduce some bias in the average estimates of the models in terms of comparison with the regional population (Bethlehem, 2010).”
ResultThe Natsal-SF: a validated measure of sexual function for use in community surveys
Mitchell, K. R., Ploubidis, G. B., Datta, J., & Wellings, K. (2012) · European Journal of Epidemiology · doi:10.1007/s10654-012-9697-3 · Cited by 81 · see citing works
“A limitation of the methodology of this study has been the use of an internet panel as a proxy to the general population [23–25].”
MethodProvincial Correctional Service Workers: The Prevalence of Mental Disorders
Carleton, R. N., Ricciardelli, R., Taillieu, T., Mitchell, M. M., Andres, E., & Afifi, T. O. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072203 · Cited by 81 · see citing works
“Third, responses were based on anonymous online self-reporting of current symptoms, making reliability and validity ambiguous [40]; however, the results are consistent with previous Canadian correctional worker responses and increased reliability and validity would require a substantial investment.”
ResultPrevalence of Lower Urinary Tract Symptoms in People with Multiple Sclerosis
Dandan, H. B. A., Coote, S. B., & McClurg, D. (2020) · International Journal of MS Care · doi:10.7224/1537-2073.2019-030 · Cited by 77 · see citing works
Apostolou, D., & Welcher, S. N. (2020) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2020.11.167 · Cited by 77 · see citing works
Uribe, S. E., Maldupa, I., Kavadella, A., El Tantawi, M., Chaurasia, A., Fontana, M., Marino, R., Innes, N., & Schwendicke, F. (2024) · European Journal of Dental Education · doi:10.1111/eje.13009 · Cited by 77 · see citing works
“The web survey method may introduce bias to the results due to potential self-selection bias, 52 which should be considered in future studies.”
MethodBrowne, J. L., Ventura, A. D., Mosely, K., & Speight, J. (2017) · Diabetic Medicine · doi:10.1111/dme.13507 · Cited by 76 · see citing works
Farmaki, A., & Kladou, S. (2020) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2020.01.005 · Cited by 79 · see citing works
An assessment of attitudes and perceptions of international university students on climate change
Filho, W. L., Ayal, D. Y., Wall, T., Shiel, C., Paço, A. M. F. D., Pace, P., Mifsud, M., Salvia, A. L., Skouloudis, A., Moggi, S., et al. (2023) · Climate Risk Management · doi:10.1016/j.crm.2023.100486 · Cited by 76 · see citing works
“This is not unusual and as the literature shows [e.g. Bethlehem, 2010], it need to be taken into account.”
ResultConsumers’ preferences for biodiversity in vineyards: A choice experiment on wine
Mazzocchi, C., Ruggeri, G., & Corsi, S. (2019) · Wine Economics and Policy · doi:10.1016/j.wep.2019.09.002 · Cited by 75 · see citing works
“The fact that the survey took place in a winery event with wine lovers participant, might be a limitation of the study and might have introduced some bias to the average estimates of the models in terms of comparisons with the all consumers’ population (Bethlehem, 2010).”
ResultConstruction supply chain risk management
Shishehgarkhaneh, M. B., Moehler, R., Fang, Y., Aboutorab, H., & Hijazi, A. A. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105396 · Cited by 75 · see citing works
Hetherington, E., Edwards, S. A., MacDonald, S. E., Racine, N. M., Madigan, S. L., McDonald, S., & Tough, S. (2021) · CMAJ Open · doi:10.9778/cmajo.20200302 · Cited by 73 · see citing works
“46 In our study, respondents were generally more affluent and more educated than nonrespondents, and had higher complete infant vaccination (85”
Bejarano‐Martín, Á., Canal‐Bedia, R., Magán‐Maganto, M., Fernández-Álvarez, C. J., Cilleros, M. V. M., Gómez, M. C. S., Primo, P. G., Rose-Sweeney, M., Boilson, A. M., Linertová, R., et al. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04253-0 · Cited by 73 · see citing works
“Therefore, the participants are usually parents and professionals who are committed in some way to autism, but also with a higher level of education, and with more economic resources relative to the general population (Bethlehem 2010; Infante-Rivard and Cusson 2018).”
BackgroundMethodSorokowska, A., Słowińska, A. M., Zbieg, A., & Sorokowski, P. (2014) · CeON Repository (Centre for Evaluation in Education and Science) · doi:10.13140/2.1.4811.5521 · Cited by 73 · see citing works
“Z drugiej strony, badacze wskazują na wiele problemów towarzyszących wykorzystaniu kwestionariuszy internetowych (Bethlehem, 2010).”
BackgroundA Meta-Analysis of How Country-Level Factors Affect Web Survey Response Rates
Daikeler, J., Silber, H., & Bošnjak, M. (2021) · International Journal of Market Research · doi:10.1177/14707853211050916 · Cited by 75 · see citing works
“Using web-based data collection methods poses methodological challenges such as non-response bias, with systematic non-response leading to errors (Bethlehem, 2010; Groves & Peytcheva, 2008; Kreuter, 2013).”
MethodService lifetime, storage time, and disposal pathways of electronic equipment: a swiss case study
Thiébaud (‐Müller), E., Hilty, L. M., Schluep, M., Widmer, R., & Faulstich, M. (2018) · Journal of Industrial Ecology · doi:10.1111/jiec.12551 · Cited by 71 · see citing works
“This means that the survey sample is not randomly selected from the target population and thus the principles of probability sampling are not followed (Bethlehem 2009, 2010; Gosling et al. 2004).”
BackgroundMethodAdamovic, M., Gahan, P. G., Olsen, J. E., Gulyas, A., Shallcross, D., & Mendoza, A. (2021) · The International Journal of Human Resource Management · doi:10.1080/09585192.2021.1913623 · Cited by 71 · see citing works
“Although we consider our sample to be representative for information and communication technology and high-technology companies, we acknowledge that respondents could have self-selected in a way that would have made our results more conservative (Bethlehem, 2010).”
BackgroundYıldırım, M., Çi̇çek, İ., & Şanlı, M. E. (2021) · Current psychology · doi:10.1007/s12144-021-01781-w · Cited by 63 · see citing works
“As such, this study may suffer from the limitation of selection bias (Bethlehem, 2010) which refers to the notion that it is completely up to participants to select themselves for the survey.”
BackgroundFerri-García, R., & Rueda, M. D. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231500 · Cited by 67 · see citing works
“This population was used originally by [44], following an experiment by [5] to measure the efficiency of adjustments in selection bias.”
Borle, P., Boerner-Zobel, F., Voelter‐Mahlknecht, S., Hasselhorn, H. M., & Ebener, M. (2020) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-020-01588-5 · Cited by 72 · see citing works
“Online surveys are particularly subject to coverage and selection bias, which can lead to unreliable survey outcomes and undermine the external validity of such studies (Bethlehem 2010).”
BackgroundWolff, W., Baumann, L., & Englert, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199554 · Cited by 65 · see citing works
“First, even though the response rate was high in comparison to similar online surveys [28,36], selection bias cannot be ruled out [37].”
ResultPatient safety culture in a university hospital
Galvão, T. F., Lopes, M. C. C., Oliva, C. C. C., Araújo, M. E. D. A., & Silva, M. T. (2018) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.2257.3014 · Cited by 64 · see citing works
“We know that recruitment of participants influences the results, especially in internet surveys(23).”
BackgroundDignity and respect during pregnancy and childbirth: a survey of the experience of disabled women
Hall, J., Hundley, V. A., Collins, B., & Ireland, J. (2018) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-018-1950-7 · Cited by 64 · see citing works
“Selection bias is a known problem with online surveys, particularly where a link is circulated to interested groups [20].”
BackgroundHromatko, I., Tonković, M., & Vranić, A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.664554 · Cited by 64 · see citing works
“Furthermore, like any online sample, ours might have been biased regarding both demographic and relevant personality characteristics – women and participants higher on conscientiousness and agreeableness are more likely to participate (e.g., Bethlehem, 2010).”
BackgroundWhere Does the Buck Stop? Ethical and Political Issues with AI in Music Creation
Morreale, F. (2021) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.86 · Cited by 64 · see citing works
Menopause knowledge and education in women under 40: Results from an online survey
Munn, C., Vaughan, L., Talaulikar, V. S., Davies, M. C., & Harper, J. C. (2022) · Women s Health · doi:10.1177/17455057221139660 · Cited by 64 · see citing works
Mental Disorder Symptoms Among Nurses in Canada
Stelnicki, A. M., & Carleton, R. (2020) · The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres · doi:10.1177/0844562120961894 · Cited by 60 · see citing works
“Recruitment methods may partially explain differences in prevalence rates between the current sample and the general population (Bethlehem, 2010); however, nurses who were experiencing the most severe symptoms may have been on leave, missed the invitation, or been too symptomatic to respond to a lengthy survey.”
BackgroundLee, Y. H., Richards, K. A. R., & Washburn, N. S. (2021) · European Review of Applied Psychology · doi:10.1016/j.erap.2021.100625 · Cited by 63 · see citing works
Delgado-Serrano, M. D. M., Melichová, K., Fadden, I. M., & Cruz-Piedrahita, C. (2024) · Land Use Policy · doi:10.1016/j.landusepol.2024.107087 · Cited by 63 · see citing works
Bar‐Zeev, Y., Shauly, M., Lee, H., & Neumark, Y. D. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18041931 · Cited by 62 · see citing works
“The sample, therefore, is likely to not be fully representative of all smokers and ex-smokers in Israel [42].”
BackgroundMaroto, M. L., Pettinicchio, D., & Lukk, M. (2021) · Sociological Perspectives · doi:10.1177/07311214211012018 · Cited by 60 · see citing works
“…population characteristics we should base any additional quotas or weights (as there have been no other random surveys of Canadians with disabilities and chronic health conditions), we do not employ poststratification weights based on group characteristics (Bethlehem 2010; Schonlau et al. 2009).”
MethodFisher, C., Mikołajczak, G., Ezer, P., Kerr, L., Bellamy, R., Brown, G., Waling, A., & Lucke, J. C. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00217 · Cited by 60 · see citing works
“Similar to many other sexual health studies, the Secondary Student Survey likely suffered a selection bias (68, 69).”
ResultLynn, P. J. (2016) · Wiley eBooks · doi:10.1002/9781118763025.ch24 · Cited by 59 · see citing works
- Authors
- Bethlehem, J.
- Affiliations
- Centraal Bureau voor de Statistiek (NL)
- Corresponding author
- Bethlehem, Jelke
- Venue
- International Statistical Review
- Publisher
- Wiley
- Published
- 2010-08
- Volume
- 78
- Issue
- 2
- Pages
- 161-188
- ISSN
- 0306-7734 (ISSN-L)
- 1751-5823
- OpenAlex
- W2129332419
- Cited by
- 1,315
- References
- 37
- Citations by year
- 2026: 228 · 2025: 173 · 2024: 151 · 2023: 149 · 2022: 122 · 2021: 144 · 2020: 109 · 2019: 47 · 2018: 38 · 2017: 54 · 2016: 20 · 2015: 32 · 2014: 19 · 2013: 14 · 2012: 6
- FWCI
- 29.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Statistical Review (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Census and Population Estimation
- Keywords
- online surveys, selection bias, undercoverage, self-selection, internet access, survey design, participant recruitment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
