The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
How Low Should You Go? Low Response Rates and the Validity of Inference in IS Questionnaire Research
Open accessAbstract
We believe IS researchers can and should do a better of job of improving (assuring) the validity of their findings by minimizing nonresponse error. To demonstrate that there is, in fact, a problem, we first present the response rates reported in six well-regarded IS journals and summarize how nonresponse error was estimated and handled in published IS research. To illustrate how nonresponse error may bias findings in IS research, we calculate its impact on confidence intervals. After demonstrating the impact of nonresponse on research findings, we discuss three post hoc remedies and three preventative measures for the IS researcher to consider. The paper concludes with a general discussion about nonresponse and its implications for IS research practice. In our delimitations section, we suggest directions for further exploring external validity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1998) · Cited by 36,049
A Theory of Cognitive Dissonance
(2017) · Macat Library eBooks · doi:10.4324/9781912282432 · Cited by 18,122
(2002) · Cited by 13,211
Estimating Nonresponse Bias in Mail Surveys
Armstrong, J. S., & Overton, T. S. (1977) · Journal of Marketing Research · doi:10.2307/3150783 · Cited by 12,222
The norm of reciprocity: A preliminary statement
Gouldner, A. W. (1960) · American Sociological Review · doi:10.2307/2092623 · Cited by 12,001
(1979) · Medical Entomology and Zoology · Cited by 11,252
Power analysis and determination of sample size for covariance structure modeling.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.130 · Cited by 10,418
Estimating nonresponse bias in mail surveys
Armstrong, J. S., & Overton, T. S. (1977) · Journal of Marketing Research · doi:10.1177/002224377701400320 · Cited by 10,390
Mail and internet surveys : the tailored design method
Dillman, D. A. (2007) · Cited by 9,587
Understanding Information Systems Continuance: An Expectation-Confirmation Model1
Bhattacherjee, A. (2001) · MIS Quarterly · doi:10.2307/3250921 · Cited by 8,767
To parcel or not to parcel: Exploring the question, weighing the merits
Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0902_1 · Cited by 6,552
Quasi-experimentation: Design and analysis issues for field settings
Shrout, P. E. (1980) · Evaluation and Program Planning · doi:10.1016/0149-7189(80)90063-4 · Cited by 6,010
Losh-Hesselbart, S., & Fowler, F. J. (1985) · Journal of the American Statistical Association · doi:10.2307/2288598 · Cited by 4,764
Influence : science and practice
Cialdini, R. B. (1985) · Cited by 3,581
Mail and Internet Surveys: The Tailored Design Method
CHISNALL, P. M. (2007) · Journal of Advertising Research · doi:10.2501/s0021849907070237 · Cited by 3,435
Validation Guidelines for IS Positivist Research
Straub, D., & Gefen, D. (2004) · Communications of the Association for Information Systems · doi:10.17705/1cais.01324 · Cited by 2,797
How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power
Muthén, L. K., & Muthén, B. O. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0904_8 · Cited by 2,362
Factors Influencing the Adoption of Internet Banking
National University of Singapore, Tan, M., Teo, T., & National University of Singapore (2000) · Journal of the Association for Information Systems · doi:10.17705/1jais.00005 · Cited by 1,716
Bandalos, D. L. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0901_5 · Cited by 1,377
The Design and Administration of Mail Surveys
Dillman, D. A. (1991) · Annual Review of Sociology · doi:10.1146/annurev.so.17.080191.001301 · Cited by 1,311
A theory of cognitive dissonance
Eysenck, H. J. (1963) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(63)90061-8 · Cited by 1,116
Alignment Between Business and IS Strategies: A Study of Prospectors, Analyzers, and Defenders
Sabherwal, R., & Chan, Y. E. (2001) · Information Systems Research · doi:10.1287/isre.12.1.11.9714 · Cited by 1,114
Response Rate in Academic Studies-A Comparative Analysis
Baruch, Y. (1999) · Human Relations · doi:10.1177/001872679905200401 · Cited by 1,112
Designing Evaluations of Educational and Social Programs.
Hendrickson, L., & Cronbach, L. J. (1983) · Contemporary Sociology · doi:10.2307/2068975 · Cited by 1,042
Armstrong, C. P., & Sambamurthy, V. (1999) · Information Systems Research · doi:10.1287/isre.10.4.304 · Cited by 1,003
Heberlein, T. A., & Baumgartner, R. (1978) · American Sociological Review · doi:10.2307/2094771 · Cited by 999
Validation in Information Systems Research: A State-of-the-Art Assessment1
Boudreau, M.-C., Gefen, D., & Straub, D. W. (2001) · MIS Quarterly · doi:10.2307/3250956 · Cited by 959
Development of a Standard E-Mail Methodology: Results of an Experiment
Schaefer, D. R., & Dillman, D. A. (1998) · Public Opinion Quarterly · doi:10.1086/297851 · Cited by 830
Understanding The Decision to Participate in a Survey
Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992) · Public Opinion Quarterly · doi:10.1086/269338 · Cited by 810
Segars, A. H., & Grover, V. (2021) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/3shh-pe41 · Cited by 806
Leverage-Saliency Theory of Survey Participation
Groves, R. M., Singer, E., & Corning, A. (2000) · Public Opinion Quarterly · doi:10.1086/317990 · Cited by 774
(2000) · Journal of Management Information Systems · doi:10.1080/07421222.2000.11045632 · Cited by 744
Research Methods in the Behavioral Sciences
Blankertz, D. F., Festinger, L., & Katz, D. (1954) · Journal of Marketing · doi:10.2307/1247742 · Cited by 708
A Quantitative Review of Research Design Effects on Response Rates to Questionnaires
Yu, J., & Cooper, H. (1983) · Journal of Marketing Research · doi:10.2307/3151410 · Cited by 706
The Concept of External Validity
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1982) · Journal of Consumer Research · doi:10.1086/208920 · Cited by 613
Online Surveys in Marketing Research
Ilieva, J., Baron, S., & Healey, N. M. (2002) · International Journal of Market Research · doi:10.1177/147078530204400303 · Cited by 595
The Search for "Optimal" Cutoff Properties: Fit Index Criteria in Structural Equation Modeling
Sivo, S. A., Fan, X., Witta, E. L., & Willse, J. T. (2006) · The Journal of Experimental Education · doi:10.3200/jexe.74.3.267-288 · Cited by 568
Self-citationUnderstanding Mail Survey Response Behavior: A Meta-Analysis
Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991) · Public Opinion Quarterly · doi:10.1086/269284 · Cited by 566
Measuring Information System Service Quality: Servqual from the Other Side1
Jiang, J. J., Klein, G., & Carr, C. L. (2002) · MIS Quarterly · doi:10.2307/4132324 · Cited by 518
Self-citationPower of the Likelihood Ratio Test in Covariance Structure Analysis
Satorra, A., & Saris, W. E. (1985) · Psychometrika · doi:10.1007/bf02294150 · Cited by 485
Interpersonal Conflict and Its Management in Information System Development1
Barki, H., & Hartwick, J. (2001) · MIS Quarterly · doi:10.2307/3250929 · Cited by 485
Exploring Perceptions of Organizational Ownership of Information and Expertise
Jarvenpaa, S. L., & Staples, D. S. (2001) · Journal of Management Information Systems · doi:10.1080/07421222.2001.11045673 · Cited by 482
On the External Validity of Experiments in Consumer Research
Lynch, Jr. (1982) · Journal of Consumer Research · doi:10.1086/208919 · Cited by 469
Interview Mode Effects in Surveys of Drug and Alcohol Use: A Field Experiment
Aquilino, W. S. (1994) · Public Opinion Quarterly · doi:10.1086/269419 · Cited by 458
A Critical Comparison of Three Strategies of Collecting Data from Households
Hochstim, J. R. (1967) · Journal of the American Statistical Association · doi:10.1080/01621459.1967.10500909 · Cited by 418
Hart, P. J., & Saunders, C. S. (1998) · Journal of Management Information Systems · doi:10.1080/07421222.1998.11518187 · Cited by 407
Self-citationStimulating Responses to Mailed Questionnaires: A Review
Linsky, A. S. (1975) · Public Opinion Quarterly · doi:10.1086/268201 · Cited by 398
Response Variation in E-Mail Surveys: An Exploration
Sheehan, K. B., & McMillan, S. J. (1999) · Journal of Advertising Research · doi:10.1080/00218499.1999.12466620 · Cited by 317
Modeling IT Ethics: A Study in Situational Ethics1
Banerjee, D., Cronan, T. P., & Jones, T. W. (1998) · MIS Quarterly · doi:10.2307/249677 · Cited by 314
The Problem of Statistical Power in MIS Research
Baroudi, J. J., & Orlikowski, W. J. (1989) · MIS Quarterly · doi:10.2307/248704 · Cited by 302
Virtual subjects: Using the Internet as an alternative source of subjects and research environment
Smith, M. A., & Leigh, B. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210601 · Cited by 288
Raising Response Rates in Mail Surveys of Small Business Owners: Results of an Experiment
Dennis jr., W. J. (2003) · Journal of Small Business Management · doi:10.1111/1540-627x.00082 · Cited by 258
Scott, C. (1961) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2984129 · Cited by 258
The Performance Impacts of Quick Response and Strategic Alignment in Specialty Retailing
Palmer, J. W., & Markus, M. L. (2000) · Information Systems Research · doi:10.1287/isre.11.3.241.12203 · Cited by 254
Jiang, J. J., & Klein, G. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518262 · Cited by 244
Self-citationResponse rates in HRM/OB survey research: Norms and correlates, 1990–1994
Roth, P. (1998) · Journal of Management · doi:10.1016/s0149-2063(99)80055-5 · Cited by 237
Tse, A. C. B. (1998) · Market Research Society. Journal. · doi:10.1177/147078539804000407 · Cited by 215
A Comparison of Four Multi-Attribute Models in the Prediction of Consumer Attitudes
Mazis, M. B., Ahtola, O. T., & Klippel, R. E. (1975) · Journal of Consumer Research · doi:10.1086/208614 · Cited by 196
External Validity and the Research Process: A Comment on the Calder/Lynch Dialogue
McGrath, J. E., & Brinberg, D. (1983) · Journal of Consumer Research · doi:10.1086/208951 · Cited by 182
Total Quality Management in Information Systems Development: Key Constructs and Relationships
Ravichandran, T., & Rai, A. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518259 · Cited by 180
Management Information Systems Research: What's There in a Methodology?
Palvia, P., Mao, E., Salam, A. F., & Soliman, K. S. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01116 · Cited by 177
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1983) · Journal of Consumer Research · doi:10.1086/208950 · Cited by 164
Mimetic Isomorphism and Technology Evaluation: Does Imitation Transcend Judgment?
University of Western Ontario, Tingling, P., Parent, M., & Simon Fraser University (2002) · Journal of the Association for Information Systems · doi:10.17705/1jais.00025 · Cited by 149
Estimating Bias Due to Nonresponse in Mail Surveys
Filion, F. L. (1975) · Public Opinion Quarterly · doi:10.1086/268245 · Cited by 147
Beyond Sabre: An Empirical Test of Expertise Exploitation in Electronic Channels1
Christiaanse, E., & Venkatraman, N. V. (2002) · MIS Quarterly · doi:10.2307/4132339 · Cited by 137
The Role of External Validity in Theoretical Research
Lynch, J. G. (1983) · Journal of Consumer Research · doi:10.1086/208949 · Cited by 126
Understanding differences in people's answers to telephone and mail surveys
Dillman, D. A., Sangster, R. L., Tarnai, J., & Rockwood, T. H. (1996) · New Directions for Evaluation · doi:10.1002/ev.1034 · Cited by 115
Cognitive Aspects of Survey Measurement and Mismeasurement
Tourangeau, R. (2003) · International Journal for Quality in Health Care · doi:10.1093/ijpor/15.1.3 · Cited by 112
A Cognitive Model of Response-Order Effects in Survey Measurement
Schwarz, N., Hippler, H., & Noëlle-Neumann, E. (1992) · Springer eBooks · doi:10.1007/978-1-4612-2848-6_13 · Cited by 101
Beyond the laboratory : field research in social psychology
Bickman, L., & Henchy, T. (1972) · McGraw-Hill eBooks · Cited by 76
Purchasing Behavior and Personal Attributes
Massy, W. F., Frank, R. E., & Lodahl, T. M. (2018) · University of Pennsylvania Press eBooks · doi:10.2307/j.ctv513btn · Cited by 71
Lowe, N. J. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511482281.003 · Cited by 66
Purchasing Behavior and Personal Attributes
Massy, W. F., Frank, R. E., & Lodahl, T. M. (1968) · University of Pennsylvania Press eBooks · doi:10.9783/9781512817850 · Cited by 65
Robins, L. N. (1963) · Public Opinion Quarterly · doi:10.1086/267168 · Cited by 64
New Approaches to Assessing Opinion: The Prospects for Electronic Mail Surveys
Best, S. J. (2002) · International Journal for Quality in Health Care · doi:10.1093/ijpor/14.1.73 · Cited by 57
de Leeuw, E. D., Mellenbergh, G. J., & Hox, J. J. (1996) · Sociological Methods & Research · doi:10.1177/0049124196024004002 · Cited by 53
Dillman, D. A., & Tarnai, J. (2004) · Wiley series in probability and statistics · doi:10.1002/9781118150382.ch5 · Cited by 48
Weighting to Adjust for Survey Nonresponse
Fuller, C. H. (1974) · Public Opinion Quarterly · doi:10.1086/268154 · Cited by 42
Nonlinear and Noncompensatory Models in User Information Satisfaction Measurement
Sethi, V., & King, R. C. (1999) · Information Systems Research · doi:10.1287/isre.10.1.87 · Cited by 42
The Materiality of Information System Planning Maturity to Project Performance
Jiang, J. J., Klein, G. L., & Shepherd, M. M. (2001) · Journal of the Association for Information Systems · doi:10.17705/1jais.00017 · Cited by 31
Self-citationDattalo, P. (2013) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199773596.003.0005 · Cited by 28
Capturing the Technological Dimensions of IT Infrastructure Change: A Model and Empirical Evidence
Shaw, N. G. (2002) · Journal of the Association for Information Systems · doi:10.17705/1jais.00020 · Cited by 26
Direction, Magnitude and Implications of Non-response Bias in Mail Surveys
Chen, H. C. K. (1996) · Market Research Society Journal · doi:10.1177/147078539603800306 · Cited by 25
Multiple Indicator Stationary Time Series Models
Sivo, S. A. (2001) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem0804_05 · Cited by 21
Self-citationElectronic Journals in Business Schools: Legitimacy, Acceptance, and Use
Palmer, J. W., Speier, C., Wren, D. A., & Hahn, S. E. (2000) · Journal of the Association for Information Systems · doi:10.17705/1jais.00002 · Cited by 15
Lei, P., & Dunbar, S. B. (2004) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1101_2 · Cited by 9
Cited by
Anderson, C. L., & Agarwal, R. (2010) · MIS Quarterly · doi:10.2307/25750694 · Cited by 713
Lee, G., & Xia, W. (2010) · MIS Quarterly · doi:10.2307/20721416 · Cited by 614
Consumers' decisions in social commerce context: An empirical investigation
Chen, J., & Shen, X.-L. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.07.012 · Cited by 493
Dincelli, E., & Yayla, A. (2022) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2022.101717 · Cited by 490
Moqbel, M., & Kock, N. (2018) · Information & Management · doi:10.1016/j.im.2017.05.001 · Cited by 269
Metadata
- Authors
- University of Central Florida, Sivo, S., Saunders, C., University of Central Florida, Chang, Q., University of Central Florida, Jiang, J., & University of Central Florida
- Venue
- Journal of the Association for Information Systems
- Publisher
- Association for Information Systems
- Published
- 2006-06
- Volume
- 7
- Issue
- 6
- Pages
- 351-414
- ISSN
- 1536-9323 (ISSN-L)
- OpenAlex
- W1533390111
- Cited by
- 376
- References
- 86
- FWCI
- 25.36
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.