The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Examination of the equivalence of self-report survey-based paper-and-pencil and internet data collection methods.
Abstract
Self-report survey-based data collection is increasingly carried out using the Internet, as opposed to the traditional paper-and-pencil method. However, previous research on the equivalence of these methods has yielded inconsistent findings. This may be due to methodological and statistical issues present in much of the literature, such as nonequivalent samples in different conditions due to recruitment, participant self-selection to conditions, and data collection procedures, as well as incomplete or inappropriate statistical procedures for examining equivalence. We conducted 2 studies examining the equivalence of paper-and-pencil and Internet data collection that accounted for these issues. In both studies, we used measures of personality, social desirability, and computer self-efficacy, and, in Study 2, we used personal growth initiative to assess quantitative equivalence (i.e., mean equivalence), qualitative equivalence (i.e., internal consistency and intercorrelations), and auxiliary equivalence (i.e., response rates, missing data, completion time, and comfort completing questionnaires using paper-and-pencil and the Internet). Study 1 investigated the effects of completing surveys via paper-and-pencil or the Internet in both traditional (i.e., lab) and natural (i.e., take-home) settings. Results indicated equivalence across conditions, except for auxiliary equivalence aspects of missing data and completion time. Study 2 examined mailed paper-and-pencil and Internet surveys without contact between experimenter and participants. Results indicated equivalence between conditions, except for auxiliary equivalence aspects of response rate for providing an address and completion time. Overall, the findings show that paper-and-pencil and Internet data collection methods are generally equivalent, particularly for quantitative and qualitative equivalence, with nonequivalence only for some aspects of auxiliary equivalence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Computer Self-Efficacy: Development of a Measure and Initial Test
Compeau, D. R., & Higgins, C. A. (1995) · MIS Quarterly · doi:10.2307/249688 · Cited by 6,553
Positive Psychology Progress: Empirical Validation of Interventions.
Seligman, M. E. P., Steen, T. A., Park, N., & Peterson, C. (2005) · American Psychologist · doi:10.1037/0003-066x.60.5.410 · Cited by 5,947
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
The international personality item pool and the future of public-domain personality measures
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2005.08.007 · Cited by 3,185
(1999) · Medical Entomology and Zoology · Cited by 3,151
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932
Schuirmann, D. J. (1987) · Journal of Pharmacokinetics and Biopharmaceutics · doi:10.1007/bf01068419 · Cited by 2,696
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,621
Bethlehem, J. (2010) · International Statistical Review · doi:10.1111/j.1751-5823.2010.00112.x · Cited by 1,315
Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis
Shih, T.-H., & Fan, X. (2008) · Field Methods · doi:10.1177/1525822x08317085 · Cited by 1,159
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999) · Journal of Applied Psychology · doi:10.1037//0021-9010.84.5.754 · Cited by 818
Social desirability, anonymity, and internet-based questionnaires
Joinson, A. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200723 · Cited by 812
van der Linden, D., te Nijenhuis, J., & Bakker, A. B. (2010) · Journal of Research in Personality · doi:10.1016/j.jrp.2010.03.003 · Cited by 598
Using the Internet for psychological research: Personality testing on the World Wide Web
Buchanan, T., & Smith, J. L. (1999) · British Journal of Psychology · doi:10.1348/000712699161189 · Cited by 594
Measuring Social Desirability: Short Forms of the Marlowe-Crowne Social Desirability Scale
Fischer, D. G., & Fick, C. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053002011 · Cited by 580
Using significance tests to evaluate equivalence between two experimental groups.
Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.113.3.553 · Cited by 566
AN EMPIRICAL ASSESSMENT OF DATA COLLECTION USING THE INTERNET
STANTON, J. M. (1998) · Personnel Psychology · doi:10.1111/j.1744-6570.1998.tb00259.x · Cited by 531
Development and Validation of the Computer Self-Efficacy Scale
Murphy, C. A., Coover, D., & Owen, S. V. (1989) · Educational and Psychological Measurement · doi:10.1177/001316448904900412 · Cited by 475
Tryon, W. W. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.371 · Cited by 391
Bäckström, M., Björklund, F., & Larsson, M. R. (2009) · Journal of Research in Personality · doi:10.1016/j.jrp.2008.12.013 · Cited by 364
Development and psychometric evaluation of the Personal Growth Initiative Scale–II.
Robitschek, C., Ashton, M. W., Spering, C. C., Geiger, N., Byers, D., Schotts, G. C., & Thoen, M. A. (2012) · Journal of Counseling Psychology · doi:10.1037/a0027310 · Cited by 359
Symmetrical Confidence Intervals for Bioequivalence Trials
Westlake, W. J. (1976) · Biometrics · doi:10.2307/2529259 · Cited by 336
E‐Research: Ethics, Security, Design, and Control in Psychological Research on the Internet
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002) · Journal of Social Issues · doi:10.1111/1540-4560.00254 · Cited by 304
Implementing a five-factor personality inventory for use on the internet
Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005) · European Journal of Psychological Assessment · doi:10.1027/1015-5759.21.2.115 · Cited by 301
Using the Internet for survey research: A case study
Zhang, Y. (2000) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(2000)51:1<57::aid-asi9>3.0.co;2-w · Cited by 295
Virtual subjects: Using the Internet as an alternative source of subjects and research environment
Smith, M. A., & Leigh, B. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210601 · Cited by 288
Barbeite, F. G., & Weiss, E. M. (2004) · Computers in Human Behavior · doi:10.1016/s0747-5632(03)00049-9 · Cited by 255
Print versus electronic surveys: a comparison of two data collection methodologies
Boyer, K. K., Olson, J. R., Calantone, R. J., & Jackson, E. C. (2002) · Journal of Operations Management · doi:10.1016/s0272-6963(02)00004-9 · Cited by 253
College Student Responses to Web and Paper Surveys: Does Mode Matter?
Carini, R. M., Hayek, J. C., Kuh, G. D., Kennedy, J. M., & Ouimet, J. A. (2003) · Research in Higher Education · doi:10.1023/a:1021363527731 · Cited by 252
The Marlowe-Crowne Affair: Short Forms, Psychometric Structure, and Social Desirability
Barger, S. D. (2002) · Journal of Personality Assessment · doi:10.1207/s15327752jpa7902_11 · Cited by 224
Web-based administration of a personality questionnaire: Comparison with traditional methods
Davis, R. N. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200737 · Cited by 223
Herrero, J., & Meneses, J. (2006) · Computers in Human Behavior · doi:10.1016/j.chb.2004.03.007 · Cited by 222
Online assessment: Desirable or dangerous?
Buchanan, T. (2002) · Professional Psychology: Research and Practice · doi:10.1037//0735-7028.33.2.148 · Cited by 210
Methodological issues in online data collection
Cantrell, M. A., & Lupinacci, P. (2007) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2007.04448.x · Cited by 208
Meyerson, P., & Tryon, W. W. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195541 · Cited by 197
When Epidemiology Meets the Internet: Web-based Surveys in the Millennium Cohort Study
Smith, B., Smith, T. C., Gray, G. C., Ryan, M. A. K., & for the Millennium Cohort Study Team (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm212 · Cited by 196
The Integrity of Web-Delivered Experiments: Can You Trust the Data?
McGraw, K. O., Tew, M. D., & Williams, J. E. (2000) · Psychological Science · doi:10.1111/1467-9280.00296 · Cited by 190
A comparison of online and postal data collection methods in marketing research
McDonald, H., & Adam, S. (2003) · Marketing Intelligence & Planning · doi:10.1108/02634500310465399 · Cited by 172
Online Versus Paper: Format Effects in Tourism Surveys
Dolnicar, S., Laesser, C., & Matus, K. (2009) · Journal of Travel Research · doi:10.1177/0047287508326506 · Cited by 171
Internet-based Questionnaire Assessment: Appropriate Use in Clinical Contexts
Buchanan, T. (2003) · Cognitive Behaviour Therapy · doi:10.1080/16506070310000957 · Cited by 168
PLOYHART, R. E., WEEKLEY, J. A., HOLTZ, B. C., & KEMP, C. (2003) · Personnel Psychology · doi:10.1111/j.1744-6570.2003.tb00757.x · Cited by 163
A comparison of World-Wide Web and paper-and-pencil personality questionnaires
Pettit, F. A. (2002) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195423 · Cited by 158
Do print and Web surveys provide the same results?
Huang, H. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2004.09.012 · Cited by 152
Insuring sample equivalence across internet and paper-and-pencil assessments
Epstein, J., Klinkenberg, W. D., Wiley, D. C., & McKinley, L. (2001) · Computers in Human Behavior · doi:10.1016/s0747-5632(01)00002-4 · Cited by 150
Comparison of Web and Mail Surveys in Collecting Illicit Drug Use Data: A Randomized Experiment
McCabe, S. E. (2004) · Journal of Drug Education · doi:10.2190/4hey-vwxl-dvr3-hakv · Cited by 147
Positive Psychology: Past, Present, and Future
Diener, E. (2009) · doi:10.1093/oxfordhb/9780195187243.013.0002 · Cited by 144
Equivalence confidence intervals for two-group comparisons of means.
Seaman, M. A., & Serlin, R. C. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.403 · Cited by 142
Fouladi, R. T., McCarthy, C. J., & Moller, N. P. (2002) · Assessment · doi:10.1177/10791102009002011 · Cited by 127
Harrison, A. W., & Rainer, R. K. (1992) · Educational and Psychological Measurement · doi:10.1177/0013164492052003024 · Cited by 126
Cronk, B. C., & West, J. L. (2002) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195440 · Cited by 126
Lewis, I. M., Watson, B. C., & White, K. M. (2008) · Australian Journal of Psychology · doi:10.1080/00049530802105865 · Cited by 124
Survey Methods in Educational Research
Berends, M. (2006) · doi:10.4324/9780203874769.ch37 · Cited by 107
Cole, M. S., Bedeian, A. G., & Feild, H. S. (2006) · Organizational Research Methods · doi:10.1177/1094428106287434 · Cited by 105
Reactions to Computerized Testing in Selection Contexts
Wiechmann, D., & Ryan, A. M. (2003) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00245 · Cited by 103
Perceived self-efficacy and technology proficiency in undergraduate college students
McCoy, C. (2010) · Computers & Education · doi:10.1016/j.compedu.2010.07.003 · Cited by 99
Buchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., & Scholey, A. B. (2005) · Behavior Research Methods · doi:10.3758/bf03206409 · Cited by 98
The null hypothesis significance-testing debate and its implications for personality research.
Fraley, R. C., & Marks, M. J. (2007) · Cited by 89
Sethuraman, R., Kerin, R. A., & Cron, W. L. (2003) · Journal of Business Research · doi:10.1016/j.jbusres.2003.09.009 · Cited by 85
Equivalence testing for use in psychosocial and services research: An introduction with examples
Stegner, B. L., Bostrom, A., & Greenfield, T. K. (1996) · Evaluation and Program Planning · doi:10.1016/0149-7189(96)00011-0 · Cited by 84
From paper to pixels: A comparison of paper and computer formats in psychological assessment
Naus, M. J., Philipp, L. M., & Samsi, M. (2008) · Computers in Human Behavior · doi:10.1016/j.chb.2008.05.012 · Cited by 80
Online- versus paper-pencil-version of a high potential intelligence test
Preckel, F., & Thiemann, H. (2003) · Swiss Journal of Psychology · doi:10.1024//1421-0185.62.2.131 · Cited by 77
Miyazaki, A. D., & Taylor, K. A. (2007) · Journal of Business Ethics · doi:10.1007/s10551-007-9547-5 · Cited by 73
Recommendations for applying tests of equivalence
Cribbie, R. A., Gruman, J. A., & Arpin‐Cribbie, C. A. (2003) · Journal of Clinical Psychology · doi:10.1002/jclp.10217 · Cited by 71
van de Vijver, F. J. R., & Harsveld, M. (1994) · Journal of Applied Psychology · doi:10.1037/0021-9010.79.6.852 · Cited by 68
Raven's Progressive Matrices (1938): More on norms, reliability, and validity
Burke, H. R. (1985) · Journal of Clinical Psychology · doi:10.1002/1097-4679(198503)41:2<236::aid-jclp2270410217>3.0.co;2-h · Cited by 65
In the Privacy of Their Own Homes: Using the Internet to Assess Racial Bias
Evans, D. C., Garcia, D. J., Garcia, D. M., & Baron, R. S. (2003) · Personality and Social Psychology Bulletin · doi:10.1177/0146167202239052 · Cited by 65
Church, A. H. (2001) · Personnel Psychology · doi:10.1111/j.1744-6570.2001.tb00238.x · Cited by 63
Assessment procedures for counselors and helping professionals
Drummond, R. J., & JONES, K. D. (2006) · Andalas University Repository (Andalas University) · Cited by 62
Wood, E., Nosko, A., Desmarais, S., Ross, C., & Irvine, C. (2006) · The Canadian Journal of Human Sexuality · Cited by 61
Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II)
Schulenberg, S. E., & Yutrzenka, B. A. (2001) · Current Psychology · doi:10.1007/s12144-001-1008-1 · Cited by 58
Steele, R. M., Mummery, W. K., & Dwyer, T. (2009) · Health Education & Behavior · doi:10.1177/1090198109335802 · Cited by 51
Schulenberg, S. E., & Yutrzenka, B. A. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207726 · Cited by 50
Hewson, C., & Charlton, J. P. (2005) · Behavior Research Methods · doi:10.3758/bf03192742 · Cited by 49
Rusticus, S. A., & Lovato, C. Y. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/d5wf-5p77 · Cited by 46
Vallejo, M. Á., Mañanes, G., Comeche, M. I., & Díaz, M. I. (2007) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2007.04.001 · Cited by 45
Whittier, D. K., Seeley, S., & Lawrence, J. S. S. (2004) · AIDS Education and Prevention · doi:10.1521/aeap.16.5.476.48735 · Cited by 38
Empirical Examination of an Online Version of the Self-Directed Search
Barak, A., & Cohen, L. (2002) · Journal of Career Assessment · doi:10.1177/1069072702238402 · Cited by 38
(2007) · WestminsterResearch (University of Westminster) · Cited by 37
Smither, J. W., Walker, A. G., & Yap, M. K. T. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403258429 · Cited by 33
Internet-based attitude assessment: does gender affect measurement equivalence?
Whitaker, B. G. (2005) · Computers in Human Behavior · doi:10.1016/j.chb.2004.11.016 · Cited by 21
Quantitative models in psychology.
McGrath, R. E. (2011) · American Psychological Association eBooks · doi:10.1037/12316-000 · Cited by 19
Patterson, S. D., & Jones, B. C. (2006) · International Journal of Pharmaceutical Medicine · doi:10.2165/00124363-200620040-00004 · Cited by 2
Development of reliable and valid short forms of the Marlowe - Crowne Social Desirability Scale
Reynolds, W. M. (1982) · Journal of Clinical Psychology · doi:10.1002/1097-4679(198201)38:1<119::aid-jclp2270380118>3.0.co;2-i
Cited by
Moshagen, M., Hilbig, B. E., & Zettler, I. (2018) · Psychological Review · doi:10.1037/rev0000111 · Cited by 519
Parsons, K., McCormac, A., Butavicius, M., Pattinson, M., & Jerram, C. (2014) · Computers & Security · doi:10.1016/j.cose.2013.12.003 · Cited by 437
Hohenberger, C., Spörrle, M., & Welpe, I. M. (2016) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2016.09.022 · Cited by 361
Social desirability is the same in offline, online, and paper surveys: A meta-analysis
Dodou, D., & de Winter, J. C. F. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.04.005 · Cited by 325
Enhancing the Validity of a Quality of Life Measure for Autistic People
McConachie, H., Mason, D., Parr, J. R., Garland, D., Wilson, C., & Rodgers, J. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-017-3402-z · Cited by 245
Metadata
- Authors
- Weigold, A., Weigold, I. K., & Russell, E. J.
- Venue
- Psychological Methods
- Published
- 2013
- Volume
- 18
- Issue
- 1
- Pages
- 53-70
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2014382881
- Cited by
- 330
- References
- 85
- FWCI
- 75.22
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.