Cited by 1,315Survey Methodology and NonresponseSurvey Sampling and Estimation TechniquesCensus and Population Estimation

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Selection Bias in Web Surveys

Open access

Bethlehem, Jelke·2010·International Statistical Review

DOI 10.1111/j.1751-5823.2010.00112.x

Open this paper in CoChat — free account Open PDF

Abstract

Summary: At first sight, web surveys seem to be an interesting and attractive means of data collection. They provide simple, cheap, and fast access to a large group of potential respondents. However, web surveys are not without methodological problems. Specific groups in the populations are under‐represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self‐selection. Both under‐coverage and self‐selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under‐coverage problems may solve itself in the future, but that self‐selection leads to unreliable survey outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1111/j.1751-5823.2010.00112.x

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,385
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008) · Cited by 5,345
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011
Web Surveys
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,086
On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086
The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies
Cochran, W. G. (1968) · Biometrics · doi:10.2307/2528036 · Cited by 961

Show all 37 references

Research works printed but not linked: 20

Bethlehem J.G., 1988, Reduction of the nonresponse bias through regression estimation, J. Official Statist., 4, 251
Bethlehem J.G., 2002, Survey Nonresponse
Bethlehem J.G., 2009, The Rise of Survey Sampling
Börsch‐Supan A., 2004, Correcting the Participation Bias in an Online Survey
Bowley A.L., 1926, Measurement of the precision attained in sampling, Bull. Int. Statist. Inst., 6
Cobben F., 2004, Nonresponse Correction Techniques in Household Surveys at Statistics Netherlands: a CAPI‐CATI Comparison
Cochran W.G., 1953, Sampling Techniques
Cochran W.G., 1977, Sampling Techniques
De Haan J., 2006, Jaarboek ICT en Samenleving, De Digitale Generatie
Eurostat, 2007, More than 40% of Households have Broadband Internet Access
Hansen M.H., 1953, Survey Sampling Methods and Theory
Kendall M.G., 1960, Where shall the history of statistics begin, Biometrika, 47, 447
Kiaer A. N., 1895, Observations et Expériences Concernant des Dénombrements Représentatives, Bull. Internat. Statist. Inst., 176
Kiaer A. N., 1997, Den Repräsentative Undersökelsesmetode, Christiania Videnskabsselskabets Skrifter. II. Historiskfilosofiske klasse, 4, 37
Kish L., 1967, Survey Sampling
Kuusela V., 2003, Proceedings of the 4th ASC International Conference, 317
10.1111/j.2397-2335.1934.tb04184.x · DOI
Scherpenzeel A., 2010, Social Research and the Internet: Advances in applied Methods and New Research Strategies
Terhanian G., 2001, Exploiting analytical advances: Minimizing the biases associated with Internet‐based surveys of non‐random samples, ESOMAR Publication Services, 248, 247
Vonk T., 2006, The effects of panel recruitment and management on research results, a study among 19 online panels, ESOMAR Publication Services, 317, 79

Sorted by citations

Items 101–200 of 1,000

The COVID-19 Psychological Wellbeing Study: Understanding the Longitudinal Psychosocial Impact of the COVID-19 Pandemic in the UK; a Methodological Overview Paper
Armour, C., McGlinchey, E., Butter, S., McAloney‐Kocaman, K., & McPherson, K. E. (2020) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-020-09841-4 · Cited by 66
“…to gather such data on a large scale amidst a pandemic and has the added benefit of increasing accessibility to those groups who would be difficult to access via other means (Wright 2005), it is also important to mention that our sampling strategy may result in self-selection bias (Bethlehem 2010).”
Background
Is there an all-embracing construct of emotion reactivity? Adaptation and validation of the emotion reactivity scale among a French-speaking community sample
Lannoy, S., Heeren, A., Rochat, L., Rossignol, M., der Linden, M. V., & Billieux, J. (2014) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2014.07.023 · Cited by 66
“Indeed, some studies [37,38] showed a selfselection bias in this kind of procedure.”
Background
Epidemiological Challenges in the Study of Behavioral Addictions: a Call for High Standard Methodologies
Rumpf, H., Brandt, D. M., Demetrovics, Z., Billieux, J., Carragher, N., Brand, M., Bowden‐Jones, H., Rahimi‐Movaghar, A., Assanangkornchai, S., Tkalić, R. G., et al. (2019) · Current Addiction Reports · doi:10.1007/s40429-019-00262-2 · Cited by 72
Changing Patterns of Substance Use During the Coronavirus Pandemic: Self-Reported Use of Tobacco, Alcohol, Cannabis, and Other Drugs
Benschop, A., van Bakkum, F., & Noijen, J. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.633551 · Cited by 60
“This study cannot claim optimal generalizability due to under-coverage and self-selection inherent to web surveys (43).”
Background
Green growth, a-growth or degrowth? Investigating the attitudes of environmental protection specialists at the German Environment Agency
Lehmann, C., Delbard, O., & Lange, S. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.130306 · Cited by 75
Paying for sustainability: A cross-cultural analysis of consumers’ valuations of food and non-food products labeled for carbon and water footprints
Grebitus, C., Steiner, B., & Veeman, M. M. (2016) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2016.05.003 · Cited by 55
Investigating selection bias of online surveys on coronavirus-related behavioral outcomes: An example utilizing the GESIS Panel Special Survey on the Coronavirus SARS-CoV-2 Outbreak in Germany
Schaurer, I., & Weiß, B. (2020) · Survey research methods · doi:10.18148/srm/2020.v14i2.7751 · Cited by 54
“…Weiß found that both selection into the survey and compliance with COVID-19 safety measures were correlated with some latent personality traits (BIG 5), which leads to biased estimates (see Bethlehem, 1988, 2010b, for a more general discussion on the issue of self-selection and biased estimates).”
BackgroundResult
The role of university switching costs, perceived service quality, perceived university image and student satisfaction in shaping student loyalty
Ali, M., Amir, H., & Ahmed, M. (2021) · Journal of Marketing for HIGHER EDUCATION · doi:10.1080/08841241.2021.1975184 · Cited by 54
“However, non-probability sampling may result in a sampling error known as selection bias (Bethlehem, 2010).”
Background
A population-based investigation of participation rate and self-selection bias in momentary data capture and survey studies
Stone, A. A., Schneider, S., Smyth, J. M., Junghaenel, D. U., Couper, M. P., Wen, C. K. F., Mendez, M., Velasco, S. E., & Goldstein, S. (2023) · Current Psychology · doi:10.1007/s12144-023-04426-2 · Cited by 56
Perceived Group Threat, Perceived Injustice, and Self-Reported Right-Wing Violence: An Integrative Approach to the Explanation Right-Wing Violence
Pauwels, L. J. R., & Heylen, B. (2017) · Journal of Interpersonal Violence · doi:10.1177/0886260517713711 · Cited by 59
“Some advantages of a web-based survey include the following: (a) costs involved are limited (Bethlehem, 2010), (b) errors in data-input are limited due to automation (Schmidt, 1997), and (c) the relative speed of data collection and storage compared with traditional paper and pencil surveys (Manfreda & Vehovar, 2008).”
Background
A systematic literature review of disability weights measurement studies: evolution of methodological choices
Charalampous, P., Polinder, S., Wothge, J., von der Lippe, E., & Haagsma, J. A. (2022) · Archives of Public Health · doi:10.1186/s13690-022-00860-z · Cited by 50
Midwives’ perceptions and experiences of engaging fathers in perinatal services
Rominov, H., Giallo, R., Pilkington, P. D., & Whelan, T. A. (2017) · Women and Birth · doi:10.1016/j.wombi.2016.12.002 · Cited by 51
Employment and the gender digital divide in Latin America: A decomposition analysis
Galperín, H., & Arcidiácono, M. (2021) · Telecommunications Policy · doi:10.1016/j.telpol.2021.102166 · Cited by 50
Individual heat adaptation: Analyzing risk communication, warnings, heat risk perception, and protective behavior in three German cities
Heidenreich, A., & Thieken, A. H. (2024) · Risk Analysis · doi:10.1111/risa.14278 · Cited by 56
“Self-selection bias may play a greater role in online surveys (Bethlehem, 2010), but since many people no longer have a landline phone, a mere telephone-based survey is not likely to represent the addressed population.”
Factors Associated With Burnout, Post-traumatic Stress and Anxio-Depressive Symptoms in Healthcare Workers 3 Months Into the COVID-19 Pandemic: An Observational Study
Cyr, S., Marcil, M.-J., Marin, M., Tardif, J., Guay, S., Guertin, M., Rosa, C., Genest, C., Forest, J., Lavoie, P., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.668278 · Cited by 58
“We must take into account another limit in the selection of our participants, as our study was Webbased (75).”
Result
Drivers of consumer willingness to pay for sustainable wines: natural, biodynamic, and organic
Vecchio, R., Annunziata, A., Parga‐Dans, E., & González, P. A. (2023) · Organic Agriculture · doi:10.1007/s13165-023-00425-6 · Cited by 51
“Third, online surveys are prone to strong sample-selection bias (Bethlehem 2010), due in this case to over-representation of individuals who are more concerned with sustainability issues.”
Background
A Systematic Review of Childhood Maltreatment Assessments in Population-Representative Surveys Since 1990
Hovdestad, W., Campeau, A., Potter, D. A., & Tonmyr, L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0123366 · Cited by 49
“However, based on sampling theory and a simulation study, Bethlehem concluded that self-selection web surveys have “no role” in creation of accurate estimates of population characteristics [126].”
Appearance, insults, allegations, blame and threats: an analysis of anonymous non-constructive student evaluation of teaching in Australia
Lakeman, R., Coutts, R., Hutchinson, M., Lee, M. F., Massey, D. L., Nasrawi, D., & Fielden, J. (2021) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2021.2012643 · Cited by 49
“There is also likely to be self-selection bias in the non-probability sampling (Bethlehem 2010) meaning that although the”
BackgroundResult
Thick Big Data
Jemielniak, D. (2020) · doi:10.1093/oso/9780198839705.001.0001 · Cited by 64
COVID-19 and the bleisure travellers: an investigation on the aftermaths and future implications
Walia, S. K., Kour, P., Choudhary, P., & Jasrotia, A. (2021) · Tourism Recreation Research · doi:10.1080/02508281.2021.1946653 · Cited by 50
The Role of Organizational Citizenship Behavior and Gender between Job Satisfaction and Task Performance
Casu, G., Mariani, M. G., Chiesa, R., Guglielmi, D., & Gremigni, P. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18189499 · Cited by 76
“Fourth, the generalizability of this study’s findings was limited by the convenience nature of our sampling strategy as well as by the use of the Internet to recruit the sample and administer the survey [76].”
Background
Sexual Minority Identities: Outness and Well-Being Among Lesbian, Gay, and Bisexual Adults
Whitman, C. N., & Nadal, K. L. Y. (2015) · Journal of Gay & Lesbian Mental Health · doi:10.1080/19359705.2015.1038974 · Cited by 58
“Furthermore, Bethlehem (2010) found that self-selection biases have the potential to be particularly influential in web-based surveys (compared with in-person participation), therefore future research should use supplemental methodologies (e.”
BackgroundMethod
Endorsement for Extremism, Exposure to Extremism via Social Media and Self-Reported Political/Religious Aggression
Pauwels, L. J. R., & Hardyns, W. (2018) · International Journal of Developmental Science · doi:10.3233/dev-170229 · Cited by 51
“Still, web surveys are increasingly accepted as a valid and reliable tool of measuring self-reported delinquency with their data quality measuring up that of paper and pencil surveys (Bethlehem, 2010).”
Use of Cigarettes and E-Cigarettes/Vaping Among Transgender People: Results From the 2015 U.S. Transgender Survey
Kcomt, L., Evans‐Polce, R. J., Veliz, P. T., Boyd, C. J., & McCabe, S. E. (2020) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2020.03.027 · Cited by 50
I did my bit! The impact of electric vehicle adoption on compensatory beliefs and norms in Norway
Nayum, A., & Thøgersen, J. B. (2022) · Energy Research & Social Science · doi:10.1016/j.erss.2022.102541 · Cited by 52
The Questionnaire of Cognitive and Affective Empathy: A Comparison Between Paper-and-Pencil Versus Online Formats in Italian Samples
Girolamo, M. D., Giromini, L., Winters, C. L., Serie, C., & de Ruiter, C. (2017) · Journal of Personality Assessment · doi:10.1080/00223891.2017.1389745 · Cited by 56
“, participants had not been personally invited by the research assistant to volunteer for the study) and undercoverage in online surveys (Bethlehem, 2010).”
Background
Being a Psychotherapist in Times of the Novel Coronavirus Disease: Stress-Level, Job Anxiety, and Fear of Coronavirus Disease Infection in More Than 1,500 Psychotherapists in Austria
Probst, T., Humer, E. C., Stippl, P., & Pieh, C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.559100 · Cited by 50
“Carrying out the survey online may also have introduced some selection bias toward fewer elder psychotherapists’ participation (Bethlehem, 2010).”
Background
High stress experienced in the foster and kin carer role: Understanding the complexities of the carer and child in context
Harding, L. F., Murray, K. E., Shakespeare‐Finch, J., & Frey, R. (2018) · Children and Youth Services Review · doi:10.1016/j.childyouth.2018.11.004 · Cited by 46
Perceived Empathy of Technology Scale (PETS): Measuring Empathy of Systems Toward the User
Schmidmaier, M., Rupp, J., Cvetanova, D., & Mayer, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642035 · Cited by 58
“This sampling strategy could potentially limit the generalizability of our fndings, and future studies should examine whether the scale is valid for other samples [10].”
People with disabilities’ perceptions of autonomous vehicles as a viable transportation option to improve mobility: An exploratory study using mixed methods
Hwang, J., Li, W., Stough, L. M., Lee, C., & Turnbull, K. F. (2020) · International Journal of Sustainable Transportation · doi:10.1080/15568318.2020.1833115 · Cited by 46
“This result may be due in part to the data collection method of this study; that is, an online survey was distributed by disability organizations and transit agencies thereby only individuals with disabilities who can access the Internet could take part in the survey (Bethlehem, 2010).”
Result
Small-scale urban agriculture: Drivers of growing produce at home and in community gardens in Detroit
Grebitus, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256913 · Cited by 48
“Nonetheless, using an online survey can cause other concerns, such as self-selection bias [89]; this risk was countered by using a relatively large sample from a local area.”
BackgroundInfluential
Self-selection and non-response biases in customers’ hotel ratings – a comparison of online and offline ratings
Smironva, E., Kiatkawsin, K., Lee, S. K., Kim, J., & Lee, C.-H. (2019) · Current Issues in Tourism · doi:10.1080/13683500.2019.1599828 · Cited by 47
“Data that is missing due to non-response might become a significant hindrance to depict accurate reporting of the product or service quality (Bethlehem, 2010; Groves, 2006; Lauw et al., 2008).”
Background
Identifying Sources of Potential Bias When Using Online Survey Data to Explore Horse Training, Management, and Behaviour: A Systematic Literature Review
Fenner, K., Hyde, M. L., Crean, A. J., & McGreevy, P. D. (2020) · Veterinary Sciences · doi:10.3390/vetsci7030140 · Cited by 45
“All online surveys are likely exposed to both self-selection bias, where only individuals with internet access who choose to visit a website and decide to participate can respond [31], and non-response bias, where participants are unwilling or unable to answer questions [32].”
Background
Nowcasting food inflation with a massive amount of online prices
Macias, P., Stelmasiak, D., & Szafranek, K. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.02.007 · Cited by 46
Understanding student behavior and perceptions toward earning badges in a gamified MOOC
Ortega-Arranz, A., Er, E., Martínez-Monés, A., Bote-Lorenzo, M., Asensio-Pérez, J., & Muñoz-Cristóbal, J. A. (2019) · Universal Access in the Information Society · doi:10.1007/s10209-019-00677-8 · Cited by 42
Measuring social licence: What and who determines public acceptability of aquaculture in New Zealand?
Sinner, J., Newton, M., Barclay, J., Baines, J. T., Farrelly, T. A., Edwards, P., & Tipa, G. T. (2020) · Aquaculture · doi:10.1016/j.aquaculture.2020.734973 · Cited by 44
Amateur and Recreational Athletes’ Motivation to Exercise, Stress, and Coping During the Corona Crisis
Lautenbach, F., Leisterer, S., Walter, N., Kronenberg, L., Manges, T., Leis, O., Pelikan, V., Gebhardt, S., & Elbe, A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.611658 · Cited by 53
“Additionally, data were assessed online and this led to selection bias in the sampling process (Bethlehem, 2010).”
Method
Investigating acceptance of marine tourism levies, to cover the opportunity costs of conservation for coastal communities
Booth, H. L., Mourato, S., & Milner‐Gulland, E. J. (2022) · Ecological Economics · doi:10.1016/j.ecolecon.2022.107578 · Cited by 43
Eliciting Egg Consumer Preferences for Organic Labels and Omega 3 Claims in Italy and Hungary
Yeh, C., Menozzi, D., & Török, Á. (2020) · Foods · doi:10.3390/foods9091212 · Cited by 43
Statewide assessment of street trees in New York State, USA
Cowett, F. D., & Bassuk, N. (2014) · Urban forestry & urban greening · doi:10.1016/j.ufug.2014.02.001 · Cited by 42
Drinking biodiversity: a choice experiment on Franciacorta sparkling wines
Ruggeri, G., Mazzocchi, C., & Corsi, S. (2020) · British Food Journal · doi:10.1108/bfj-06-2019-0451 · Cited by 42
Self-reported vs Directly Observed Face Mask Use in Kenya
Jakubowski, A., Egger, D. T., Nekesa, C., Lowe, L., Walker, M., & Miguel, E. A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.18830 · Cited by 41
Impact of COVID-19 pandemic on anxiety, depression and distress – online survey results amid the pandemic in Russia
Karpenko, O., Syunyakov, T., Kulygina, M. A., Pavlichenko, A. V., Chetkina, A. S., & Andrushchenko, A. (2020) · Consortium Psychiatricum · doi:10.17650/2712-7672-2020-1-1-8-20 · Cited by 40
“The main limitation of our survey is the unknown selection bias [27].”
Background
Pet Reptiles—Are We Meeting Their Needs?
Azevedo, A., Guimarães, L., Ferraz, J., Whiting, M. C., & Magalhães‐Sant’Ana, M. (2021) · Animals · doi:10.3390/ani11102964 · Cited by 40
Active versus passive: evaluating the effectiveness of inoculation techniques in relation to misinformation about climate change
Green, M., McShane, C. J., & Swinbourne, A. L. (2022) · Australian Journal of Psychology · doi:10.1080/00049530.2022.2113340 · Cited by 47
“…not to mention that this study was potentially limited by the recruitment strategy of utilising social media platforms which may lead to selfselection bias as inherently only those with access to the internet and those who self-select to participate in the survey are captured (Bethlehem, 2010).”
Background
Where does knowledge-sharing motivation come from? The case of third-party developer in mobile platforms
Choi, G., Nam, C., Kim, S., Jung, H. J., & Lee, C. H. (2020) · Journal of Knowledge Management · doi:10.1108/jkm-08-2019-0449 · Cited by 41
Street Tree Diversity in Three Northeastern U.S. States
Cowett, F. D., & Bassuk, N. (2017) · Arboriculture & Urban Forestry · doi:10.48044/jauf.2017.001 · Cited by 40
The New Internet World: A Global Perspective on Freedom of Expression, Privacy, Trust and Security OnlinePreprint
Dutta, S., Dutton, W. H., & Law, G. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1916005 · Cited by 40
Measuring COVID-19 Related Anxiety in Parents: Psychometric Comparison of Four Different Inventories
Kubb, C., & Foran, H. M. (2020) · JMIR Mental Health · doi:10.2196/24507 · Cited by 43
British Public Opinion after a Decade of War: Attitudes to Iraq and Afghanistan
Gribble, R., Wessely, S. C., Klein, S. J., Alexander, D. M., Dandeker, C., & Fear, N. T. (2014) · Politics · doi:10.1111/1467-9256.12073 · Cited by 39
“However, there are potential issues regarding the representativeness of samples in relation to some social groups, such as those without Internet or landline access (Bethlehem, 2010; Blumberg and Luke, 2009; Duffy and Smith, 2005).”
Background
The Influence of a Person’s Digital Affinity on Unit Nonresponse and Attrition in an Online Panel
Herzing, J. M. E., & Blom, A. G. (2018) · Social Science Computer Review · doi:10.1177/0894439318774758 · Cited by 38
Consequences of information suppression in ecological and conservation sciences
Driscoll, D. A., Garrard, G. E., Kusmanoff, A. M., Dovers, S. R., Maron, M., Preece, N., Pressey, R. L., & Ritchie, E. G. (2020) · Conservation Letters · doi:10.1111/conl.12757 · Cited by 43
“Not being based on a probability sample, the results cannot be used to infer the proportion of the ecological community who have experienced constraints on communication of information (Bethlehem, 2010).”
Global UN 2030 agenda: How can Science, Technology and Innovation accelerate the achievement of Sustainable Development Goals for All?
Adenle, A. A., Steur, H. D., Mwongera, C., Rola‐Rubzen, M. F., de Barcellos, M. D., Vivanco, D. F., Timilsina, G. R., Possas, C. D. A., Alders, R., Chertow, M. R., et al. (2023) · PLOS Sustainability and Transformation · doi:10.1371/journal.pstr.0000085 · Cited by 37
Chronic pain among public safety personnel in Canada
Carleton, R. N., Afifi, T. O., Turner, S. E., Taillieu, T., El-Gabalawy, R. M., Sareen, J., & Asmundson, G. J. G. (2017) · Canadian Journal of Pain · doi:10.1080/24740527.2017.1410431 · Cited by 39
Evaluation of a citizen science platform for collecting fisheries data from coastal sea trout anglers
Gundelund, C., Venturelli, P., Hartill, B., Hyder, K., Olesen, H. J., & Skov, C. (2021) · Canadian Journal of Fisheries and Aquatic Sciences · doi:10.1139/cjfas-2020-0364 · Cited by 37
“A common approach (e.g., when comparing self-selected web panels to probabilistic surveys) is to present point estimates from the raw unweighted data, such as the means (e.g., Bethlehem 2010, 2015).”
Will COVID-19 accelerate telecommuting? A cross-country evaluation for Israel and Czechia
Kogus, A., Foltýnová, H. B., Gal‐Tzur, A., Shiftan, Y., Vejchodská, E., & Shiftan, Y. (2022) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2022.08.011 · Cited by 38
“An additional methodological caveat had been addressed by Bethlehem (2010) and Currie et al. (2021), who point out that online surveys have a notable and significant “under-representation bias” for lower income and older age groups.”
How Do Clinical Information Systems Affect the Cognitive Demands of General Practitioners?: Usability Study with a Focus on Cognitive Workload
Ariza, F., Kalra, D., & Potts, H. W. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i4.85 · Cited by 34
Do German hospital report cards have the potential to improve the quality of care?
Emmert, M., Hessemer, S., Meszmer, N., & Sander, U. (2014) · Health Policy · doi:10.1016/j.healthpol.2014.07.006 · Cited by 37
A Structural Equation Model of Perceived Autonomy Support and Growth Mindset in Undergraduate Students: The Mediating Role of Sense of Coherence
Ma, C., Ma, Y., & Lan, X. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02055 · Cited by 39
“, simple and easy to operate, affordable, and fast access to the possible participants) and the increasing popularity of using the online-based survey in psychology research, it also involves some potential drawbacks (e.g., under-coverage and self-selection bias; Bethlehem, 2010).”
Background
Understanding Consumers’ Preferences for Protected Geographical Indications: A Choice Experiment with Hungarian Sausage Consumers
Török, Á., Gorton, M., Yeh, C., Czine, P., & Balogh, P. (2022) · Foods · doi:10.3390/foods11070997 · Cited by 36
“However, middle-aged, better-educated and urban respondents are overrepresented, which is typical for only online surveys [61].”
Background
Uncertainty about the risks associated with microplastics among lay and topic-experienced respondents
Thiele, C. J., & Hudson, M. D. (2021) · Scientific Reports · doi:10.1038/s41598-021-86569-5 · Cited by 39
Purchase Intentions in a Chatbot Environment: An Examination of the Effects of Customer Experience
Martínez Puertas, S., Illescas Manzano, M. D., Segovia López, C., & Ribeiro-Cardoso, P. (2024) · Oeconomia Copernicana · doi:10.24136/oc.2914 · Cited by 34
“Although, convenience samples have been questioned in marketing research (Ashraf & Merunka, 2017) due to the lack of generalizability and the related issues of non-random selection (Bethlehem, 2010), several reasons can justify their use.”
Background
A dynamic capability perspective on the impact of big data analytics and enterprise architecture on innovation: an empirical study
Pathak, S., Krishnaswamy, V., & Sharma, M. K. (2024) · Journal of Enterprise Information Management · doi:10.1108/jeim-01-2024-0059 · Cited by 44
What Makes a Rabbit Cute? Preference for Rabbit Faces Differs according to Skull Morphology and Demographic Factors
Harvey, N. D., Oxley, J. A., Miguel-Pacheco, G. G., Gosling, E. M., & Farnworth, M. J. (2019) · Animals · doi:10.3390/ani9100728 · Cited by 36
“As a web-survey largely distributed through social media, the data described here were collected through self-selection, which introduces its own form of bias [50].”
Method
The tourist experience in the digital era: the case of Italian millennials
Pencarelli, T., Gabbianelli, L., & Savelli, E. (2021) · Sinergie Italian Journal of Management · doi:10.7433/s113.2020.10 · Cited by 34
FORUM: Remote testing for psychological and physiological acoustics
Peng, Z. E., Waz, S., Buss, E., Shen, Y., Richards, V. M., Bharadwaj, H., Stecker, G. C., Beim, J. A., Bosen, A. K., Braza, M. D., et al. (2022) · The Journal of the Acoustical Society of America · doi:10.1121/10.0010422 · Cited by 35
“Self-selection may, therefore, result in certain populations being underrepresented in the participant sample, leading to lower generalizability in the research findings (Bethlehem, 2010; Turner et al., 2020).”
Result
Social communication and emotion difficulties and second to fourth digit ratio in a large community-based sample
Barona, M., Kothari, R., Skuse, D., & Micali, N. (2015) · Molecular Autism · doi:10.1186/s13229-015-0063-7 · Cited by 34
“However, this study has methodological shortcomings, including self-measurement of the finger lengths, which has been shown to be unreliable [54] (a recent meta-analysis estimated the reliability of the measurement to be 46 % of that of expert-measured 2D:4D) [17]; reliance on a subset of not validated questions to as…”
Method
Quantifying perceived impact of scientific publications
Radicchi, F., Weissman, A., & Bollen, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.05.010 · Cited by 33
Validation of a revised General Nutrition Knowledge Questionnaire for Australia
Thompson, C., Vidgen, H. A., Gallegos, D., & Hannan-Jones, M. T. (2020) · Public Health Nutrition · doi:10.1017/s1368980019005135 · Cited by 35
“questionnaire was developed online, it may not be appropriate for all demographics or socio435 economic groups within Australia due to issues associated with internet access, and self436 selection methods((70)).”
Assessing the cost and quality-of-life impact of on-demand‐only medications for adults with hereditary angioedema
Castaldo, A. J., Jervelund, C., Corcoran, D., Boysen, H. B., Christiansen, S. C., & Zuraw, B. L. (2021) · Allergy and Asthma Proceedings · doi:10.2500/aap.2021.42.200127 · Cited by 33
Relationships among behavioural regulations, physical activity, and mental health pre- and during COVID–19 UK lockdown
Bird, J. M., Karageorghis, C. I., & Hamer, M. (2021) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2021.101945 · Cited by 35
“Non-probability sampling was used and there is a participant self-selection bias that is common to surveys of this nature, meaning that lower socio-economic groups and ethnic minorities are underrepresented (Bethlehem, 2010; Spence et al., 2020).”
Method
Are multi-cat homes more stressful? A critical review of the evidence associated with cat group size and wellbeing
Finka, L. R., & Foreman-Worsley, R. (2021) · Journal of Feline Medicine and Surgery · doi:10.1177/1098612x211013741 · Cited by 34
Between Consultation and Collaboration: Self-Reported Objectives for 25 Web-Based Geoparticipation Projects in Urban Planning
Babelon, I., Pánek, J., Falco, E., Kleinhans, R., & Charlton, J. I. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi10110783 · Cited by 33
Are crypto and non-crypto investors alike? Evidence from a comprehensive survey in Brazil
Colombo, J. A., & Yarovaya, L. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102468 · Cited by 32
COVID-19: Guaranteed Loans and Zombie Firms
Zoller-Rydzek, B., & Keller, F. (2020) · CESifo Economic Studies · doi:10.1093/cesifo/ifaa014 · Cited by 32
The living, the dead, and the obsolete: A characterization of lifetime and stock of ICT products in Denmark
Zhilyaev, D., Cimpan, C., Cao, Z., Liu, G., Askegaard, S., & Wenzel, H. (2020) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2020.105117 · Cited by 32
Incidence of PTSD and generalized anxiety symptoms during the first wave of COVID-19 outbreak: an exploratory study of a large sample of the Italian population
Brivio, E., Oliveri, S., Guiddi, P., & Pravettoni, G. (2021) · BMC Public Health · doi:10.1186/s12889-021-11168-y · Cited by 36
“This obviously limits the scope of the hypothesis here presented and may bring to biased estimates [58].”
Background
Are Canadians more willing to provide chest-compression-only cardiopulmonary resuscitation (CPR)?—a nation-wide public survey
Cheskes, L., Morrison, L. J., Beaton, D., Parsons, J. A., & Dainty, K. N. (2015) · Canadian Journal of Emergency Medicine · doi:10.1017/cem.2015.113 · Cited by 32
Antidepressants, withdrawal, and addiction; where are we now?
Jauhar, S., Hayes, J., Goodwin, G. M., Baldwin, D. S., Cowen, P. J., & Nutt, D. J. (2019) · Journal of Psychopharmacology · doi:10.1177/0269881119845799 · Cited by 31
“Though undoubtedly rich in qualitative data, these convenience sample surveys are not generally accepted as quantitative analyses with clear criteria set by journals for inclusion of quantitative data (Bethlehem, 2010; Cook et al., 2000).”
The Effect of Survey Mode on Data Quality: Disentangling Nonresponse and Measurement Error Bias
Felderer, B., Kirchner, A., & Kreuter, F. (2019) · Journal of Official Statistics · doi:10.2478/jos-2019-0005 · Cited by 31
“Another option is to rely on different weighting techniques (Bethlehem 2010).”
Background
Surveillance of CKD epidemiology in the US – a joint analysis of NHANES and KEEP
Myers, O., Pankratz, V. S., Norris, K. C., Vassalotti, J. A., Unruh, M. L., & Argyropoulos, C. (2018) · Scientific Reports · doi:10.1038/s41598-018-34233-w · Cited by 31
“The use of such a reference, random probability sample is an integral 166 component of existing approaches for handling selection bias(13,17,20,21).”
BackgroundMethod
Riding Soundness—Comparison of Subjective With Objective Lameness Assessments of Owner-Sound Horses at Trot on a Treadmill
Müller-Quirin, J., Dittmann, M. T., Roepstorff, C., Arpagaus, S., Latif, S. N., & Weishaupt, M. A. (2020) · Journal of Equine Veterinary Science · doi:10.1016/j.jevs.2020.103314 · Cited by 31
“has previously been shown that online surveys are prone to a selfselection bias of the respondents [28].”
Background
Understanding channel choice in users' reporting behavior: Evidence from a smart mobility case
Müller, A. P. R., Lerusse, A., Steen, T., & de Walle, S. V. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101540 · Cited by 28
Self-Selection and Misreporting in Legislative Surveys
Bundi, P., Varone, F., Gava, R., & Widmer, T. (2016) · Political Science Research and Methods · doi:10.1017/psrm.2016.35 · Cited by 31
Income inequality in the United States and its potential effect on oral health
Moeller, J., Starkel, R., Quiñonez, C., & Vujicic, M. (2017) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2017.02.052 · Cited by 25
Eating disorder examination questionnaire (EDE-Q): validity and norms for Saudi nationals
Melisse, B., van Furth, E. F., & de Beurs, E. (2022) · Eating and Weight Disorders - Studies on Anorexia, Bulimia and Obesity · doi:10.1007/s40519-021-01150-3 · Cited by 35
Health Professional Advice and Adult Action to Reduce Sodium Intake
Jackson, S. L., King, S. M. C., Park, S.-Y., Fang, J., Odom, E., & Cogswell, M. E. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2015.04.034 · Cited by 30
“In another online survey, 78% of respondents reported they had taken at least one of six specified actions to limit sodium consumption,(14) although it may be that prompting with examples increased positive responses,(15) or that participants responding to an online survey about nutrition may be interested in nutritio…”
Cohort profiles of the cross-sectional and prospective participant groups in the second Diabetes MILES—Australia (MILES-2) study
Browne, J. L., Holmes‐Truscott, E., Ventura, A. D., Hendrieckx, C., Pouwer, F., & Speight, J. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012926 · Cited by 31
“Main language spoken at home—English 2759 (97) 446 (98) NS”
Background
Motivations behind users’ participation in the standard-setting process: Focus on financial analysts
Allini, A., Aria, M., Macchioni, R., & Zagaria, C. (2018) · Journal of Accounting and Public Policy · doi:10.1016/j.jaccpubpol.2018.04.002 · Cited by 30
New graduate physiotherapists’ perceived preparedness for clinical practice. A cross-sectional survey
Chesterton, P., Chesterton, J., & Alexanders, J. (2021) · European Journal of Physiotherapy · doi:10.1080/21679169.2021.1958007 · Cited by 36
“It is possible that the respondents who completed the online questionnaire from the invitations are not representative of the entire target population [84,85].”
The Role of Physicians in Digitalizing Health Care Provision: Web-Based Survey Study
Burmann, A., Tischler, M., Faßbach, M., Schneitler, S., & Meister, S. (2021) · JMIR Medical Informatics · doi:10.2196/31527 · Cited by 32
“With the first contact point being the digital channels of professional networks, a selection bias can be assumed [49].”
Background
Pervasive Uncertainty under Threat: Mental Health Disorders and Experiences of Uncertainty for Correctional Workers
Ricciardelli, R., Mitchell, M. M., Taillieu, T., Angehrn, A., Afifi, T. O., & Carleton, R. N. (2021) · Criminal Justice and Behavior · doi:10.1177/00938548211050112 · Cited by 35
“Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161–188. Blevins, C. A., Weathers, F. W., Davis, M. T., Witte, T. k., & Domino, J. L. (2015). The posttraumatic stress disorder checklist for DSM-5 (PCL-5): Development and initial psychometric evaluation. Journal of Traumati…”
BackgroundInfluential
Saving for old age: Longevity annuity buying intention of Italian young adults
Nosi, C., D’Agostino, A., Pagliuca, M. M., & Pratesi, C. A. (2014) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2014.05.001 · Cited by 32
The need for improved emotional support: a pilot online survey of Australian women’s access to healthcare services and support at the time of miscarriage
Bilardi, J. E., Sharp, G., Payne, S., & Temple-Smith, M. J. (2021) · Women and Birth · doi:10.1016/j.wombi.2020.06.011 · Cited by 29
Exposure to asylum seekers and changing support for the radical right: A natural experiment in the Netherlands
Tolsma, J., Laméris, J., & Savelkoul, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0245644 · Cited by 28
“Because people sign up for this online panel of their own, there is a self-selection bias in the sample of respondents [52].”
Background
2-step Gradient Boosting approach to selectivity bias correction in tax audit: an application to the VAT gap in Italy
Loro, P. A. D., Scacciatelli, D., & Tagliaferri, G. (2022) · Statistical Methods & Applications · doi:10.1007/s10260-022-00643-4 · Cited by 40
“It is applied in different contexts, e.g.: when incorporating the response probability to correct for the nonresponse bias (Bethlehem (1988), Alho (1990), Bethlehem (2010)), in the case of the inverse probability of treatment weighting (IWTP) (Hirano et al. (2003), Austin and Stuart (2015)).”
Artificial Intelligence (AI) integration in Rural Philippine Higher Education: Perspectives, challenges, and ethical considerations
Villarino, R. T. (2025) · IJERI: International Journal of Educational Research and Innovation · doi:10.46661/ijeri.10909 · Cited by 29
Experiencing virtual geographic environment in urban 3D participatory e-planning: A user perspective
Chassin, T., Ingensand, J., Christophe, S., & Touya, G. (2022) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2022.104432 · Cited by 29
Is the web a promising tool for data collection in developing countries? An analysis of the sample bias of 10 web and face-to-face surveys from Africa, Asia, and South America
Tijdens, K. G., & Steinmetz, S. (2015) · International Journal of Social Research Methodology · doi:10.1080/13645579.2015.1035875 · Cited by 29
“In particular, the increasing use of web surveys has continuously fueled a discussion about their quality and reliability for scientific use (Bethlehem, 2010; Couper, 2000; Fricker & Schonlau, 2002; Heerwegh & Loosveldt, 2009; Taylor, 2005; Tuten, Urban, & Bosnjak, 2002).”
Web based health surveys: Using a Two Step Heckman model to examine their potential for population health analysis
Morrissey, K., Kinderman, P., Pontin, E. E., Tai, S., & Schwannauer, M. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.06.053 · Cited by 29

Authors: Bethlehem, J.
Affiliations: Centraal Bureau voor de Statistiek (NL)
Corresponding author: Bethlehem, Jelke
Venue: International Statistical Review
Publisher: Wiley
Published: 2010-08
Volume: 78
Issue: 2
Pages: 161-188
ISSN: 0306-7734 (ISSN-L)
1751-5823
DOI: 10.1111/j.1751-5823.2010.00112.x
OpenAlex: W2129332419
Cited by: 1,315
References: 37
Citations by year: 2026: 228 · 2025: 173 · 2024: 151 · 2023: 149 · 2022: 122 · 2021: 144 · 2020: 109 · 2019: 47 · 2018: 38 · 2017: 54 · 2016: 20 · 2015: 32 · 2014: 19 · 2013: 14 · 2012: 6
FWCI: 29.95
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations: International Statistical Review (publishedVersion)
UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
RePEc: Research Papers in Economics (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics: Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
Physical Sciences › Mathematics › Statistics and Probability › Census and Population Estimation
Keywords: online surveys, selection bias, undercoverage, self-selection, internet access, survey design, participant recruitment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Selection Bias in Web Surveys

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks