The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Selection Bias in Web Surveys
Open accessAbstract
Summary: At first sight, web surveys seem to be an interesting and attractive means of data collection. They provide simple, cheap, and fast access to a large group of potential respondents. However, web surveys are not without methodological problems. Specific groups in the populations are under‐represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self‐selection. Both under‐coverage and self‐selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under‐coverage problems may solve itself in the future, but that self‐selection leads to unreliable survey outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The reliability and validity of web-based self-reported mental disorder symptom clusters remains ambiguous,71 certainly for the current population”
From Mental Disorder Symptoms among Public Safety Personnel in Canada (2018) · cites this paper as [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, the current rates may be inflated because of a self-selection bias of motivated responding by clinically symptomatic persons.71”
From Mental Disorder Symptoms among Public Safety Personnel in Canada (2018) · cites this paper as [71]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,385
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008) · Cited by 5,345
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,086
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961
Research works printed but not linked: 20
- Bethlehem J.G., 1988, Reduction of the nonresponse bias through regression estimation, J. Official Statist., 4, 251
- Bethlehem J.G., 2002, Survey Nonresponse
- Bethlehem J.G., 2009, The Rise of Survey Sampling
- Börsch‐Supan A., 2004, Correcting the Participation Bias in an Online Survey
- Bowley A.L., 1926, Measurement of the precision attained in sampling, Bull. Int. Statist. Inst., 6
- Cobben F., 2004, Nonresponse Correction Techniques in Household Surveys at Statistics Netherlands: a CAPI‐CATI Comparison
- Cochran W.G., 1953, Sampling Techniques
- Cochran W.G., 1977, Sampling Techniques
- De Haan J., 2006, Jaarboek ICT en Samenleving, De Digitale Generatie
- Eurostat, 2007, More than 40% of Households have Broadband Internet Access
- Hansen M.H., 1953, Survey Sampling Methods and Theory
- Kendall M.G., 1960, Where shall the history of statistics begin, Biometrika, 47, 447
- Kiaer A. N., 1895, Observations et Expériences Concernant des Dénombrements Représentatives, Bull. Internat. Statist. Inst., 176
- Kiaer A. N., 1997, Den Repräsentative Undersökelsesmetode, Christiania Videnskabsselskabets Skrifter. II. Historiskfilosofiske klasse, 4, 37
- Kish L., 1967, Survey Sampling
- Kuusela V., 2003, Proceedings of the 4th ASC International Conference, 317
- 10.1111/j.2397-2335.1934.tb04184.x · DOI
- Scherpenzeel A., 2010, Social Research and the Internet: Advances in applied Methods and New Research Strategies
- Terhanian G., 2001, Exploiting analytical advances: Minimizing the biases associated with Internet‐based surveys of non‐random samples, ESOMAR Publication Services, 248, 247
- Vonk T., 2006, The effects of panel recruitment and management on research results, a study among 19 online panels, ESOMAR Publication Services, 317, 79
Cited by
Sorted by citations
Expectations of brilliance underlie gender distributions across academic disciplines
Leslie, S.-J., Cimpian, A., Meyer, M., & Freeland, E. (2015) · Science · doi:10.1126/science.1261375 · Cited by 1,516
Mental Disorder Symptoms among Public Safety Personnel in Canada
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Duranceau, S., LeBouthillier, D. M., Sareen, J., Ricciardelli, R., MacPhee, R. S., Groll, D., et al. (2018) · The Canadian Journal of Psychiatry · doi:10.1177/0706743717723825 · Cited by 558
“In contrast, the current rates may be inflated because of a self-selection bias of motivated responding by clinically symptomatic persons.(71) That argument might be countered by notions that particularly symptomatic persons may also be less likely to participate in research.”
BackgroundThe role of urban green space for human well-being
Bertram, C., & Katrin Rehdanz, A. (2015) · Ecological Economics · doi:10.1016/j.ecolecon.2015.10.013 · Cited by 515
Associations between green/blue spaces and mental health across 18 countries
White, M. P., Elliott, L. R., Grellier, J., Economou, T., Bell, S., Bratman, G. N., Cirach, M., Gascón, M., Lima, M. L., Lõhmus, M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-87675-0 · Cited by 557
Krieger, N. (2020) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094017 · Cited by 411
“, mainly highly educated and affluent persons) (18, 22, 201).”
BackgroundSocial Media, Web, and Panel Surveys: Using Non‐Probability Samples in Social and Policy Research
Lehdonvirta, V., Oksanen, A., Räsänen, P., & Blank, G. (2020) · Policy & Internet · doi:10.1002/poi3.238 · Cited by 459
Bono, S. A., Villela, E. F. D. M., Siau, C. S., Chen, W. S., Pengpid, S., Hasan, M. T., Sessou, P., Ditekemena, J., Amodan, B. O., Hosseinipour, M. C., et al. (2021) · Vaccines · doi:10.3390/vaccines9050515 · Cited by 350
“Therefore, weights were applied to the study data [48].”
MethodInference for Nonprobability Samples
Elliott, M. R., & Valliant, R. (2017) · Statistical Science · doi:10.1214/16-sts598 · Cited by 331
“Bethlehem (2010) reviews this issue for web surveys. Vonk, van Ossenbruggen and Willems (2006) report that ethnic minorities and immigrant groups were systematically underrepresented in Dutch panels. They also found that, relative to the general population, the Dutch online panels contained disproportionately more vot…”
BackgroundInfluentialWeigold, A., Weigold, I. K., & Russell, E. J. (2013) · Psychological Methods · doi:10.1037/a0031607 · Cited by 379
Hoffkling, A., Obedin‐Maliver, J., & Sevelius, J. M. (2017) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-017-1491-5 · Cited by 327
“As other studies have shown, participants engaging in online studies are disproportionately educated and economically secure [24].”
ResultThe use of sampling methods in advertising research: a gap between theory and practice
Sarstedt, M., Bengart, P., Shaltoni, A. M., & Lehmann, S. (2017) · International Journal of Advertising · doi:10.1080/02650487.2017.1348329 · Cited by 369
New Developments in Survey Data Collection
Couper, M. P. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053613 · Cited by 253
Busch, I. M., Moretti, F., Purgato, M., Barbui, C., Wu, A. W., & Rimondini, M. (2019) · Journal of Patient Safety · doi:10.1097/pts.0000000000000589 · Cited by 231
Service quality in a higher education context: an integrated model
Sultan, P., & Wong, H. Y. (2012) · Asia Pacific Journal of Marketing and Logistics · doi:10.1108/13555851211278196 · Cited by 255
“Thus, a random technique was adopted (Bryman and Bell, 2007; Bethlehem, 2010).”
MethodKhazaal, Y., van Singer, M., Chatton, A., Achab, S., Zullino, D., Rothen, S., Khan, R., Billieux, J., & Thorens, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2759 · Cited by 201
“It is difficult to estimate the impact of any selection bias because information on non-participants is usually not available, and comparisons between the included and the excluded samples are not feasible [12].”
BackgroundResultInfluentialRadojević, T., Stanišić, N., & Stanić, N. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.04.002 · Cited by 194
What’s to Like? Facebook as a Tool for Survey Data Collection
Schneider, D., & Harknett, K. (2019) · Sociological Methods & Research · doi:10.1177/0049124119882477 · Cited by 192
“The weight of early research and discussion suggested that problems of undercoverage from limited and selective Internet usage made this approach of limited use (Best, Krueger, and Hubbard 2001; Bethlehem 2010; Yeager et al. 2011).”
BackgroundPerils and Potentials of Self-Selected Entry to Epidemiological Studies and Surveys
Keiding, N., & Louis, T. A. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12136 · Cited by 186
Kelly, A., Sullivan, M., & Strampel, K. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.6.12 · Cited by 177
“Finally, the authors acknowledge the general limitations of online surveys with respect to self-selection bias and potentially limited internet access of target participants during the time in which the survey was conducted (Bethlehem, 2010).”
Impacts of Covid-19 on peer-to-peer accommodation platforms: Host perceptions and responses
Farmaki, A., Miguel, C., Drotarova, M. H., Aleksić, A., Časni, A. Č., & Efthymiadou, F. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102663 · Cited by 196
“In order to minimise self-selection bias (Bethlehem, 2010), the researchers tried to ensure that the sample from each country was diverse enough (Ritchie et al., 2014) in terms of gender, age and host type.”
Estimating Propensity Adjustments for Volunteer Web Surveys
Valliant, R. L., & Dever, J. A. (2011) · Sociological Methods & Research · doi:10.1177/0049124110392533 · Cited by 167
“The model can be a simple one where means differ among groups, leading to the poststra-tified estimator (e.g., see Bethlehem 2010), or a more elaborate one involving multiple covariates, as used by Dever, Rafferty, and Valliant (2008).”
MethodCurrie, G., Jain, T., & Aston, L. (2021) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2021.09.009 · Cited by 164
Russo, D., Hanel, P. H. P., Altnickel, S., & van Berkel, N. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09945-9 · Cited by 196
“In sum, it is a convenient, fair, and efficient way to recruit survey informants (Bethlehem 2010).”
BackgroundLiu, Y., Huang, K., Bao, J., & Chen, K. (2018) · Tourism Management · doi:10.1016/j.tourman.2018.10.004 · Cited by 162
What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 166
“In our survey, self-selection bias may have also influenced our results [3]: our respondents may feel more strongly about pain points in notebooks than the general population of data scientists.”
ResultThe Importance of Selection Bias in Internet Surveys
Greenacre, Z. A. (2016) · Open Journal of Statistics · doi:10.4236/ojs.2016.63035 · Cited by 147
Suicidal ideation, plans, and attempts among public safety personnel in Canada
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., LeBouthillier, D. M., Duranceau, S., Sareen, J., Ricciardelli, R., MacPhee, R. S., Groll, D., et al. (2018) · Canadian Psychology / Psychologie canadienne · doi:10.1037/cap0000136 · Cited by 151
“Relatedly, the reliability and validity of Web based self-reports relative to clinician administered interviews remains ambiguous (Bethlehem, 2010).”
BackgroundReport of the AAPOR task force on non-probability sampling
Baker, R., Brick, J. M., Bates, N., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013) · Cited by 151
Mental health training, attitudes toward support, and screening positive for mental disorders
Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Vaughan, A. D., Anderson, G. S., Ricciardelli, R., MacPhee, R. S., Cramm, H. A., Czarnuch, S., et al. (2020) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2019.1575900 · Cited by 135
Patterns of Implicit and Explicit Attitudes: IV. Change and Stability From 2007 to 2020
Charlesworth, T. E. S., & Banaji, M. R. (2022) · Psychological Science · doi:10.1177/09567976221084257 · Cited by 138
“After all, as a convenience sample, the data set can suffer from selection bias (Bethlehem, 2010).”
BackgroundSuspicious and fraudulent online survey participation: Introducing the REAL framework
Lawlor, J., Thomas, C. C., Guhin, A. T., Kenyon, K., Lerner, M. D., & Drahota, A. (2021) · Methodological Innovations · doi:10.1177/20597991211050467 · Cited by 167
“…include self-selection bias among participants, under-coverage of individuals without Internet access, and participants’ ability to misrepre-sent their eligibility to participate or mask their identities in order to participate multiple times in a single study (Bethlehem, 2010; Wright, 2005).”
Remote smartphone monitoring of Parkinson’s disease and individual response to therapy
Omberg, L., Neto, E. C., Perumal, T. M., Pratap, A., Tediarjo, A., Adams, J., Bloem, B. R., Bot, B. M., Elson, M. J., Goldman, S. M., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00974-9 · Cited by 141
Fibromyalgia: a new facet of the post-COVID-19 syndrome spectrum? Results from a web-based survey
Ursini, F., Ciaffi, J., Mancarella, L., Lisi, L., Brusi, V., Cavallari, C., D’Onghia, M., Mari, A., Borlandelli, E., Cordella, J. F., et al. (2021) · RMD Open · doi:10.1136/rmdopen-2021-001735 · Cited by 135
Krouwel, A., Kutiyski, Y., van Prooijen, J.-W., Martinsson, J., & Markstedt, E. (2017) · Journal of Social and Political Psychology · doi:10.5964/jspp.v5i2.745 · Cited by 137
“Datasets collected online generally suffer from problems of under-coverage and self-selection that can potentially bias estimates (Bethlehem, 2010; Hooghe & Teepe, 2007).”
Meule, A., Müller, A., Gearhardt, A. N., & Jens Blechert, A. (2016) · Appetite · doi:10.1016/j.appet.2016.10.003 · Cited by 136
Influencing factors for the purchase intention of consumers choosing bioplastic products in Germany
Klein, F., Emberger‐Klein, A., Menrad, K., Möhring, W., & Blesin, J.-M. (2019) · Sustainable Production and Consumption · doi:10.1016/j.spc.2019.01.004 · Cited by 136
How the European debt crisis reshaped national political space: The case of Greece
Κατσανίδου, Α., & Otjes, S. (2015) · European Union Politics · doi:10.1177/1465116515616196 · Cited by 135
“Like most online surveys, VAA samples suffer from undercoverage and self-selection that could potentially bias estimates (Bethlehem, 2010).”
BackgroundBruckner, S., Wilson, M., Aurell, D., Rennich, K., vanEngelsdorp, D., Steinhauer, N., & Williams, G. R. (2023) · Journal of Apicultural Research · doi:10.1080/00218839.2022.2158586 · Cited by 134
“…employ different data collection and reporting approaches which harbor their own advantages and disadvantages (Evans & Mathur, 2005); for example, while online questionnaires provide the benefit of a wide geographical reach, they disadvantage beekeepers without internet access (Bethlehem, 2010).”
BackgroundInformation Could Reduce Consumer Demand for Exotic Pets
Moorhouse, T. P., Balaskas, M., D’Cruze, N. C., & Macdonald, D. W. (2016) · Conservation Letters · doi:10.1111/conl.12270 · Cited by 133
“Our study was limited to respondents with internet access (Bethlehem 2010) and whose Web browsers searched for English language websites.”
BackgroundInfluentialA multi-group analysis of convenience samples: free, cheap, friendly, and fancy sources
Winton, B. G., & Sabol, M. A. (2021) · International Journal of Social Research Methodology · doi:10.1080/13645579.2021.1961187 · Cited by 159
“We understand issues of non-random selection and assignment inherent with the use of either traditional or online convenience sampling techniques (Bethlehem, 2010; Shadish et al., 2002).”
BackgroundMethodStatistical challenges of administrative and transaction data
Hand, D. J. (2018) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12315 · Cited by 133
“De Veaux and Hand (2005) gave examples of data errors and their consequences, and national statistical institutes often define several dimensions of quality, including accuracy, relevance, timeliness, existence, coherence, completeness, accessibility and security (see, for example, Eurostat (2000) and the archives of…”
BackgroundPsychometric Properties of the Greek Version of FCV-19S
Tsipropoulou, V., Nikopoulou, V. A., Holeva, V., Nassika, Z., Diakogiannis, I., Sakka, S., Kostikidou, S., Varvara, C., Spyridopoulou, E., & Parlapani, E. (2020) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-020-00319-8 · Cited by 127
Morse, K. F., Fine, P. A., & Friedlander, K. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.609967 · Cited by 148
“As the survey was administered online, response rates were likely affected by self-selection biases, that is, completed by those interested in leisure during COVID-19 or who had additional time to do so (Bethlehem, 2010).”
MethodKawamura, T., Mori, T., Motonishi, T., & Ogawa, K. (2021) · Journal of the Japanese and International Economies · doi:10.1016/j.jjie.2021.101131 · Cited by 161
Kim, B., & Kim, M. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100590 · Cited by 150
Who mothers mommy? Factors that contribute to mothers’ well-being.
Luthar, S. S., & Ciciolla, L. M. (2015) · Developmental Psychology · doi:10.1037/dev0000051 · Cited by 143
“Limitations and caveats Given our web-based data collection, there remains a possibility of some sampling biases (Bethlehem, 2010), but at least one major source of such bias – lack of access to computers – is unlikely to have been a major problem given our focus on upper middle class mothers.”
BackgroundSurvey Weights: A Step-by-step Guide to Calculation
Dever, J. A., & Valliant, R. L. (2017) · Cited by 115
Pettinicchio, D., Maroto, M. L., Chai, L., & Lukk, M. (2021) · Disability and health journal · doi:10.1016/j.dhjo.2021.101085 · Cited by 114
COVID-19 Vaccination Acceptance and Hesitancy among Healthcare Workers in Germany
Holzmann‐Littig, C., Braunisch, M. C., Kranke, P., Popp, M., Seeber, C., Fichtner, F., Littig, B., Carbajo-Lozoya, J., Allwang, C., Frank, T., et al. (2021) · Vaccines · doi:10.3390/vaccines9070777 · Cited by 114
“Selection bias is a common problem in online surveys [58], which cannot be ruled out.”
BackgroundGoing web or staying paper? The use of web-surveys among older people
Kelfve, S., Kivi, M., Johansson, B. E. A., & Lindwall, M. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01138-0 · Cited by 114
Cheah, J., Roldán, J. L., Ciavolino, E., Ting, H., & Ramayah, T. (2020) · Total Quality Management & Business Excellence · doi:10.1080/14783363.2020.1754125 · Cited by 116
“Finally, researchers should engage in different ex-post adjustments of sampling unit weights to compensate for critical issues in survey research, such as unequal selection probabilities, non-coverage, and sampling fluctuations (Bethlehem, 2010; Sarstedt et al., 2018a).”
BackgroundTobacco Use Patterns in Five Countries During the COVID-19 Lockdown
Yach, D. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa097 · Cited by 105
Mental health in chronic disease patients during the COVID-19 quarantine in Greece
Louvardi, M., Pelekasis, P., Chrousos, G. P., & Darviri, C. (2020) · Palliative & Supportive Care · doi:10.1017/s1478951520000528 · Cited by 122
Humer, E. C., Stippl, P., Pieh, C., Pryss, R. C., & Probst, T. (2020) · Journal of Medical Internet Research · doi:10.2196/20246 · Cited by 117
“The web-based conduction of the survey may also have caused a selection bias toward participation by fewer older psychotherapists [40].”
BackgroundRestar, A. (2019) · Archives of Sexual Behavior · doi:10.1007/s10508-019-1453-2 · Cited by 114
“…is a type of bias that, when introduced in the survey, motivates participants (i.e., parental-respondents) to elect themselves into the study such that this selection is different from the study enrollment criteria (Bethlehem, 2010; Cozby, 2012; DelgadoRodriguez & Llorca, 2004; Ruane, 2005).”
ResultVannini, P., Gagliardi, G. P., Kuppe, M. K., Dossett, M. L., Donovan, N. J., Gatchel, J. R., Quiroz, Y. T., Premnath, P. Y., Amariglio, R. E., Sperling, R. A., et al. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.03.050 · Cited by 107
“We acknowledge that specific groups in the population may be under-represented as they may not have an email address and/or have limited or no access to internet (Bethlehem 2010).”
BackgroundInfluentialMobile Research Methods: Opportunities and challenges of mobile research methodologies
Toninelli, D., Pintér, R., & de Pedraza, P. (2015) · Ubiquity Press eBooks · doi:10.5334/bar · Cited by 98
The power of storytelling and video: a visual rhetoric for science communication
Finkler, W., & León, B. (2019) · Journal of Science Communication · doi:10.22323/2.18050202 · Cited by 105
“The research design had some limitations that should be kept in perspective when considering the results: (i) while the online survey provided a cost- and time-effective means of garnering a large sample size, the online and social media nature of its distribution introduced a self-selection bias, whereby survey compl…”
BackgroundResultWhat it feels like to be a mother: Variations by children’s developmental stages.
Luthar, S. S., & Ciciolla, L. M. (2015) · Developmental Psychology · doi:10.1037/dev0000062 · Cited by 105
“There remains a possibility of some sampling biases given our web-based data-collection (Bethlehem, 2010), but at least one major source of such bias – lack of access to computers – is unlikely to have been a serious problem given our targeted sample of upper middle class mothers.”
BackgroundOptions for Conducting Web Surveys
Schonlau, M., & Couper, M. P. (2017) · Statistical Science · doi:10.1214/16-sts597 · Cited by 95
“Bias and variance are reduced if both conditions hold (Bethlehem, 2010).”
BackgroundMondon, A. (2022) · Politics · doi:10.1177/02633957221104726 · Cited by 94
“…crucial to note that my interest here is not to engage with methodological issues regarding the collection of data, which have been well covered already (see, among others, Bernstein et al., 2001; Bethlehem, 2010; Bon et al., 2018; Harbaugh, 1996; Karp and Brockington, 2005; Morton et al., 2015).”
Heijnen, M., & Rietdyk, S. (2015) · Human Movement Science · doi:10.1016/j.humov.2015.12.007 · Cited by 93
Warren, A. M., Zolfaghari, K., Fresnedo, M., Bennett, M., Pogue, J. R., Waddimba, A. C., Zvolensky, M. J., Carlbring, P., & Powers, M. B. (2021) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2021.1874505 · Cited by 89
“Although our sample size was large, we relied on the use of Qualtrics patient panels thus the sample may have some bias towards individuals who self-select to be participants for online survey research (Bethlehem, 2010; Bhole & Hanna, 2017; Greenacre, 2016; Søgaard et al., 2004).”
MethodZhao, H., He, X., Fan, G., Li, L., Huang, Q., Qiu, Q.-M., Kang, Z., Du, T., Han, L., Ding, L., et al. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.085 · Cited by 88
“Therefore, this survey may suffer from the problems of under-coverage and selfselection (Bethlehem, 2010; Greenacre, 2016).”
Psychological distress and coping styles in teachers: A preliminary study
Stapleton, P., Garby, S., & Sabot, D. (2020) · Australian Journal of Education · doi:10.1177/0004944120908960 · Cited by 102
“A second limitation involves the self-selection of the convenience sample, which can lead to biased estimates (Bethlehem, 2010).”
Moeller, J., Brackett, M. A., Ivčević, Z., & White, A. E. (2020) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.101301 · Cited by 102
Hwang, J., Kim, I., & Gulzar, M. A. (2020) · Sustainability · doi:10.3390/su12041440 · Cited by 84
“In particular, as a cross-cultural testing is very important for environmental research [54] it is necessary to verify the research model proposed in this study based on the cross-cultural test in future studies; Second, this paper focused on the context of DFDS, so the results of this study could be different if the…”
MethodDemartini, E., Vecchiato, D., Tempesta, T., Gaviglio, A., & Viganò, R. (2018) · Meat Science · doi:10.1016/j.meatsci.2018.07.031 · Cited by 83
“The authors acknowledge that this might be a limitation of the study and might introduce some bias in the average estimates of the models in terms of comparison with the regional population (Bethlehem, 2010).”
ResultThe Natsal-SF: a validated measure of sexual function for use in community surveys
Mitchell, K. R., Ploubidis, G. B., Datta, J., & Wellings, K. (2012) · European Journal of Epidemiology · doi:10.1007/s10654-012-9697-3 · Cited by 81
“A limitation of the methodology of this study has been the use of an internet panel as a proxy to the general population [23–25].”
MethodProvincial Correctional Service Workers: The Prevalence of Mental Disorders
Carleton, R. N., Ricciardelli, R., Taillieu, T., Mitchell, M. M., Andres, E., & Afifi, T. O. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072203 · Cited by 96
“Third, responses were based on anonymous online self-reporting of current symptoms, making reliability and validity ambiguous [40]; however, the results are consistent with previous Canadian correctional worker responses and increased reliability and validity would require a substantial investment.”
ResultPrevalence of Lower Urinary Tract Symptoms in People with Multiple Sclerosis
Dandan, H. B. A., Coote, S. B., & McClurg, D. (2020) · International Journal of MS Care · doi:10.7224/1537-2073.2019-030 · Cited by 78
Apostolou, D., & Welcher, S. N. (2020) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2020.11.167 · Cited by 77
Uribe, S. E., Maldupa, I., Kavadella, A., El Tantawi, M., Chaurasia, A., Fontana, M., Marino, R., Innes, N., & Schwendicke, F. (2024) · European Journal of Dental Education · doi:10.1111/eje.13009 · Cited by 77
“The web survey method may introduce bias to the results due to potential self-selection bias, 52 which should be considered in future studies.”
MethodBrowne, J. L., Ventura, A. D., Mosely, K., & Speight, J. (2017) · Diabetic Medicine · doi:10.1111/dme.13507 · Cited by 76
Farmaki, A., & Kladou, S. (2020) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2020.01.005 · Cited by 79
An assessment of attitudes and perceptions of international university students on climate change
Filho, W. L., Ayal, D. Y., Wall, T., Shiel, C., Paço, A. M. F. D., Pace, P., Mifsud, M., Salvia, A. L., Skouloudis, A., Moggi, S., et al. (2023) · Climate Risk Management · doi:10.1016/j.crm.2023.100486 · Cited by 76
“This is not unusual and as the literature shows [e.g. Bethlehem, 2010], it need to be taken into account.”
ResultConsumers’ preferences for biodiversity in vineyards: A choice experiment on wine
Mazzocchi, C., Ruggeri, G., & Corsi, S. (2019) · Wine Economics and Policy · doi:10.1016/j.wep.2019.09.002 · Cited by 87
“The fact that the survey took place in a winery event with wine lovers participant, might be a limitation of the study and might have introduced some bias to the average estimates of the models in terms of comparisons with the all consumers’ population (Bethlehem, 2010).”
ResultConstruction supply chain risk management
Shishehgarkhaneh, M. B., Moehler, R., Fang, Y., Aboutorab, H., & Hijazi, A. A. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105396 · Cited by 75
Hetherington, E., Edwards, S. A., MacDonald, S. E., Racine, N. M., Madigan, S. L., McDonald, S., & Tough, S. (2021) · CMAJ Open · doi:10.9778/cmajo.20200302 · Cited by 73
“46 In our study, respondents were generally more affluent and more educated than nonrespondents, and had higher complete infant vaccination (85”
Bejarano‐Martín, Á., Canal‐Bedia, R., Magán‐Maganto, M., Fernández-Álvarez, C. J., Cilleros, M. V. M., Gómez, M. C. S., Primo, P. G., Rose-Sweeney, M., Boilson, A. M., Linertová, R., et al. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04253-0 · Cited by 88
“Therefore, the participants are usually parents and professionals who are committed in some way to autism, but also with a higher level of education, and with more economic resources relative to the general population (Bethlehem 2010; Infante-Rivard and Cusson 2018).”
BackgroundMethodSorokowska, A., Słowińska, A. M., Zbieg, A., & Sorokowski, P. (2014) · CeON Repository (Centre for Evaluation in Education and Science) · doi:10.13140/2.1.4811.5521 · Cited by 73
“Z drugiej strony, badacze wskazują na wiele problemów towarzyszących wykorzystaniu kwestionariuszy internetowych (Bethlehem, 2010).”
BackgroundA Meta-Analysis of How Country-Level Factors Affect Web Survey Response Rates
Daikeler, J., Silber, H., & Bošnjak, M. (2021) · International Journal of Market Research · doi:10.1177/14707853211050916 · Cited by 75
“Using web-based data collection methods poses methodological challenges such as non-response bias, with systematic non-response leading to errors (Bethlehem, 2010; Groves & Peytcheva, 2008; Kreuter, 2013).”
MethodService lifetime, storage time, and disposal pathways of electronic equipment: a swiss case study
Thiébaud (‐Müller), E., Hilty, L. M., Schluep, M., Widmer, R., & Faulstich, M. (2018) · Journal of Industrial Ecology · doi:10.1111/jiec.12551 · Cited by 71
“This means that the survey sample is not randomly selected from the target population and thus the principles of probability sampling are not followed (Bethlehem 2009, 2010; Gosling et al. 2004).”
BackgroundMethodAdamovic, M., Gahan, P. G., Olsen, J. E., Gulyas, A., Shallcross, D., & Mendoza, A. (2021) · The International Journal of Human Resource Management · doi:10.1080/09585192.2021.1913623 · Cited by 71
“Although we consider our sample to be representative for information and communication technology and high-technology companies, we acknowledge that respondents could have self-selected in a way that would have made our results more conservative (Bethlehem, 2010).”
BackgroundYıldırım, M., Çi̇çek, İ., & Şanlı, M. E. (2021) · Current Psychology · doi:10.1007/s12144-021-01781-w · Cited by 73
“As such, this study may suffer from the limitation of selection bias (Bethlehem, 2010) which refers to the notion that it is completely up to participants to select themselves for the survey.”
BackgroundFerri-García, R., & Rueda, M. D. M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231500 · Cited by 67
“This population was used originally by [44], following an experiment by [5] to measure the efficiency of adjustments in selection bias.”
Borle, P., Boerner-Zobel, F., Voelter‐Mahlknecht, S., Hasselhorn, H. M., & Ebener, M. (2020) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-020-01588-5 · Cited by 72
“Online surveys are particularly subject to coverage and selection bias, which can lead to unreliable survey outcomes and undermine the external validity of such studies (Bethlehem 2010).”
BackgroundWolff, W., Baumann, L., & Englert, C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199554 · Cited by 65
“First, even though the response rate was high in comparison to similar online surveys [28,36], selection bias cannot be ruled out [37].”
ResultPatient safety culture in a university hospital
Galvão, T. F., Lopes, M. C. C., Oliva, C. C. C., Araújo, M. E. D. A., & Silva, M. T. (2018) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.2257.3014 · Cited by 64
“We know that recruitment of participants influences the results, especially in internet surveys(23).”
BackgroundDignity and respect during pregnancy and childbirth: a survey of the experience of disabled women
Hall, J., Hundley, V. A., Collins, B., & Ireland, J. (2018) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-018-1950-7 · Cited by 64
“Selection bias is a known problem with online surveys, particularly where a link is circulated to interested groups [20].”
BackgroundHromatko, I., Tonković, M., & Vranić, A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.664554 · Cited by 64
“Furthermore, like any online sample, ours might have been biased regarding both demographic and relevant personality characteristics – women and participants higher on conscientiousness and agreeableness are more likely to participate (e.g., Bethlehem, 2010).”
BackgroundWhere Does the Buck Stop? Ethical and Political Issues with AI in Music Creation
Morreale, F. (2021) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.86 · Cited by 64
Menopause knowledge and education in women under 40: Results from an online survey
Munn, C., Vaughan, L., Talaulikar, V. S., Davies, M. C., & Harper, J. C. (2022) · Women s Health · doi:10.1177/17455057221139660 · Cited by 71
Mental Disorder Symptoms Among Nurses in Canada
Stelnicki, A. M., & Carleton, R. (2020) · The Canadian journal of nursing research = Revue canadienne de recherche en sciences infirmieres · doi:10.1177/0844562120961894 · Cited by 60
“Recruitment methods may partially explain differences in prevalence rates between the current sample and the general population (Bethlehem, 2010); however, nurses who were experiencing the most severe symptoms may have been on leave, missed the invitation, or been too symptomatic to respond to a lengthy survey.”
BackgroundLee, Y. H., Richards, K. A. R., & Washburn, N. S. (2021) · European Review of Applied Psychology · doi:10.1016/j.erap.2021.100625 · Cited by 74
Delgado-Serrano, M. D. M., Melichová, K., Fadden, I. M., & Cruz-Piedrahita, C. (2024) · Land Use Policy · doi:10.1016/j.landusepol.2024.107087 · Cited by 74
Bar‐Zeev, Y., Shauly, M., Lee, H., & Neumark, Y. D. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18041931 · Cited by 62
“The sample, therefore, is likely to not be fully representative of all smokers and ex-smokers in Israel [42].”
BackgroundMaroto, M. L., Pettinicchio, D., & Lukk, M. (2021) · Sociological Perspectives · doi:10.1177/07311214211012018 · Cited by 62
“…population characteristics we should base any additional quotas or weights (as there have been no other random surveys of Canadians with disabilities and chronic health conditions), we do not employ poststratification weights based on group characteristics (Bethlehem 2010; Schonlau et al. 2009).”
MethodFisher, C., Mikołajczak, G., Ezer, P., Kerr, L., Bellamy, R., Brown, G., Waling, A., & Lucke, J. C. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00217 · Cited by 60
“Similar to many other sexual health studies, the Secondary Student Survey likely suffered a selection bias (68, 69).”
ResultLynn, P. J. (2016) · Wiley eBooks · doi:10.1002/9781118763025.ch24 · Cited by 59
Metadata
- Authors
- Bethlehem, J.
- Affiliations
- Centraal Bureau voor de Statistiek (NL)
- Corresponding author
- Bethlehem, Jelke
- Venue
- International Statistical Review
- Publisher
- Wiley
- Published
- 2010-08
- Volume
- 78
- Issue
- 2
- Pages
- 161-188
- ISSN
- 0306-7734 (ISSN-L)
- 1751-5823
- OpenAlex
- W2129332419
- Cited by
- 1,315
- References
- 37
- Citations by year
- 2026: 228 · 2025: 173 · 2024: 151 · 2023: 149 · 2022: 122 · 2021: 144 · 2020: 109 · 2019: 47 · 2018: 38 · 2017: 54 · 2016: 20 · 2015: 32 · 2014: 19 · 2013: 14 · 2012: 6
- FWCI
- 29.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Statistical Review (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Census and Population Estimation
- Keywords
- online surveys, selection bias, undercoverage, self-selection, internet access, survey design, participant recruitment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
